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Abstract 

We present a new algorithm for probabilistic planning with no observability. Our algorithm, 
called Probabilistic-FF, extends the heuristic forward-search machinery of Conformant-FF to prob- 
lems with probabilistic uncertainty about both the initial state and action effects. Specifically, 
Probabilistic-FF combines Conformant-FF' s techniques with a powerful machinery for weighted 
model counting in (weighted) CNFs, serving to elegantly define both the search space and the 
heuristic function. Our evaluation of Probabilistic-FF shows its fine scalability in a range of prob- 
abilistic domains, constituting a several orders of magnitude improvement over previous results in 
this area. We use a problematic case to point out the main open issue to be addressed by further 
research. 

1. Introduction 

In this paper we address the problem of probabilistic planning with no observability (Kushmerick, 
Hanks, & Weld, 1995), also known in the AI planning community as conditional (Majercik & 
Littman, 2003) or conformant (Hyafil & Bacchus, 2004) probabilistic planning. In such problems 
we are given an initial belief state in the form of a probability distribution over the world states W , 
a set of actions (possibly) having probabilistic effects, and a set of alternative goal states Wq ^ W. 
A solution to such a problem is a single sequence of actions that transforms the system into one 
of the goal states with probability exceeding a given threshold 6. The basic assumption of the 
problem is that the system cannot be observed at the time of plan execution. Such a setting is 
useful in controlling systems with uncertain initial state and non-deterministic actions, if sensing is 
expensive or unreliable. Non-probabilistic conformant planning may fail due to non-existence of a 
plan that achieves the goals with 100% certainty. Even if there is such a plan, that plan does not 
necessarily contain information about what actions are most useful to achieve only the requested 
threshold 9. 

The state-of-the-art performance of probabilistic planners has been advancing much more slowly 
than that of deterministic planners, scaling from 5-10 step plans for problems with ?a20 world states 
to 15-20 step plans for problems with R^lOO world states (Kushmerick et al., 1995; Majercik & 
Littman, 1998; Hyafil & Bacchus, 2004). Since probabilistic planning is inherently harder than its 
deterministic counterpart (Littman, Goldsmith, & Mundhenk, 1998), such a difference in evolution 
rates is by itself not surprising. However, recent developments in the area (Onder, Whelan, & Li, 
2006; Bryce, Kambhampati, & Smith, 2006; Huang, 2006), and in particular our work here, show 
that dramatic improvements in probabiUstic planning can be obtained. 
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In this paper we introduce Probabilistic-FF, a new probabilistic planner based on heuristic for- 
ward search in the space of implicitly represented probabilistic belief states. The planner is a natural 
extension of the recent (non-probabilistic) conformant planner Conformant-FF (Hoffmann & Braf- 
man, 2006). The main trick is to replace Conformant-FF' s SAT-based techniques with a recent 
powerful technique for probabilistic reasoning by weighted model counting (WMC) in preposi- 
tional CNFs (Sang, Beame, & Kautz, 2005). In more detail, Conformant-FF does a forward search 
in a belief space in which each belief state corresponds to a set of world states considered to be 
possible. The main trick of Conformant-FF is the use of CNF formulas for an impUcit represen- 
tation of beUef states. ImpUcit, in this context, means that formulas ^(a) encode the semantics of 
executing action sequence a in the initial belief state, with propositional variables corresponding to 
facts with time-stamps. Any actual knowledge about the behef states has to be (and can be) inferred 
from these formulas. Most particularly, a fact p is known to be true in a belief state if and only if 
(f)(ia) p{m), where m is the time endpoint of the formula. The only knowledge computed by 
Conformant-FF about belief states are these known facts, as well as (symmetrically) the facts that 
are known to be false. This suffices to do STRIPS-style planning, that is, to determine apphcable 
actions and goal belief states. In the heuristic function, FF's (Hoffmann & Nebel, 2001) relaxed 
planning graph technique is enriched with approximate SAT reasoning. 

The basic ideas underlying ProbabiUstic-FF are: 

(i) Define time-stamped Bayesian networks (BNs) describing probabilistic behef states. 

(ii) Extend Conformant-FF' s belief state CNFs to model these BNs. 

(iii) In addition to the SAT reasoning used by Conformant-FF, use weighted model-counting to 
determine whether the probability of the (unknown) goals in a belief state is high enough. 

(iv) Introduce approximate probabihstic reasoning into Conformant-FF's heuristic function. 

Note the synergetic effect: Probabilistic-FF re-uses all of Conformant-FF's technology to recognize 
facts that are true or false with probabihty 1. This fully serves to determine apphcable actions, as 
well as detect whether part of the goal is already known. In fact, it is as if Conformant-FF's CNF- 
based techniques were specifically made to suit the probabilistic setting: while without probabilities 
one could imagine successfully replacing the CNFs with BDDs, with probabilities this seems much 
more problematic. 

The algorithms we present cover probabihstic initial behef states given as Bayesian networks, 
deterministic and probabilistic actions, conditional effects, and standard action preconditions. Our 

experiments show that our approach is quite effective in a range of domains. In contrast to the 
SAT and CSP based approaches mentioned above (Majercik & Littman, 1998; Hyafil & Bacchus, 
2004), Probabihstic-FF can find 100-step plans for problem instances with billions of world states. 
However, such a comparison is not entirely fair due to the different nature of the results provided; the 
SAT and CSP based approaches provide guarantees on the length of the solution. The approach most 
closely related to Probabilistic-FF is implemented in POND (Bryce et al., 2006): this system, like 
Probabilistic-FF, does conformant probabilistic planning for a threshold 9, using a non-admissible, 
planning-graph based heuristic to guide the search. Hence a comparison between Probabilistic-FF 
and POND is fair, and in our experiments we perform a comparative evaluation of Probabilistic-FF 
and POND. While the two approaches are related, there are significant differences in the search 
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space representation, as well as in the definition and computation of tlie lieuristic function. We run 
the two approaches on a range of domains partly taken from the probabilistic planning literature, 
partly obtained by enriching conformant benchmarks with probabihties, and partly obtained by 
enriching classical benchmarks with probabilistic uncertainty. In almost all cases, Conformant-FF 
outperforms POND by at least an order of magnitude. We make some interesting observations 
regarding the behavior of the two planners; in particular we identify a domain - derived from the 
classical Logistics domain - where both approaches fail to scale. The apparent reason is that neither 
approach is good enough at detecting how many times, at an early point in the plan, a probabilistic 
action must be applied in order to sufficiently support a high goal threshold at the end of the plan. 
Devising methods that are better in this regard is the most pressing open issue in this line of work. 

The paper is structured as follows. The next section provides the technical background, formally 
defining the problem we address and illustrating it with our running example. Section 3 details how 
probabilistic belief states are represented as time-stamped Bayesian networks, how these Bayesian 
networks are encoded as weighted CNF formulas, and how the necessary reasoning is performed 
on this representation. Section 4 explains and illustrates our extension of Conformant-FF's heuris- 
tic function to the probabilistic settings. Section 5 provides the empirical results, and Section 6 
concludes. All proofs are moved into Appendix A. 

2. Background 

The probabilistic planning framework we consider adds probabilistic uncertainty to a subset of 
the classical ADL language, namely (sequential) STRIPS with conditional effects. Such STRIPS 
planning tasks are described over a set of propositions V as triples {A, /, G), corresponding to the 
action set, initial world state, and goals. I and G are sets of propositions, where / describes a 
concrete initial state wi, while G describes the set of goal states w ^ G. Actions a are pairs 
{pre{a), E{a)) of the precondition and the (conditional) effects. A conditional effect e is a triple 
{con{e),add{e),del{e)) of (possibly empty) proposition sets, corresponding to the effect's con- 
dition, add, and delete Usts, respectively. The precondition pre{a) is also a proposition set, and 
an action a is applicable in a world state w if w ^ pre(a). If a is not applicable in w, then 
the result of applying a to w is undefined. If a is applicable in w, then all conditional effects 
e € E{a) with w 2 con(e) occur. Occurrence of a conditional effect e inw results in the world 
state w U add{e) \ del{e). 

If an action a is applied to w, and there is a proposition q such that q e add{e) n del{e') for 
(possibly the same) occurring e, e' € E{a), then the result of applying a in w is undefined. Thus, 
we require the actions to be not self-contradictory, that is, for each a ^ A, and every e, e' € E{a), 
if there exists a world state w 2 con{e) U con{e'), then add{e) n del{e') = 0. Finally, an action 
sequence a is a plan if the world state that results from iterative execution of a's actions, starting in 
wi, leads to a goal state w D G. 

2.1 Probabilistic Planning 

Our probabilistic planning setting extends the above with (i) probabilistic uncertainty about the 
initial state, and (ii) actions that can have probabilistic effects. In general, probabilistic planning 

1. POND does not use implicit belief states, and the probabilistic part of its heuristic function uses sampling techniques, 
rather than the probabilistic reasoning techniques we employ. 
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tasks are quadruples {A,bi,G,6), corresponding to the action set, initial belief state, goals, and 
acceptable goal satisfaction probability. As before, G is a set of propositions. The initial state is 
no longer assumed to be known precisely. Instead, we are given a probability distribution over the 
world states, bj, where hi{w) describes the likelihood of w being the initial world state. 

Similarly to classical planning, actions a G A are pairs (pre(a), E{a)), but the effect set E{a) 
for such o has richer structure and semantics. Each e G E{a) is a pair (con(e), A(e)) of a propo- 
sitional condition and a set of probabilistic outcomes. Each probabihstic outcome e G A(e) is a 
triplet {Pr{e), add{e), del{e)), where add and delete lists are as before, and Pr{£) is the proba- 
biUty that outcome e occurs as a result of effect e. Naturally, we require that probabilistic effects 
define probability distributions over their outcomes, that is, Yl,eeMe) ^^i^) = 1- The special case of 
deterministic effects e is modeled this way via A(e) = {s} and Pr{£) = 1. Unconditional actions 
are modeled as having a single effect e with con(e) = 0. As before, if a is not applicable in w, 
then the result of applying a to wis undefined. Otherwise, if a is applicable in w, then there exists 
exactly one effect e G E{a) such that con{e) C u;, and for each e G A(e), applying a to it; results 
in w U add{e) \ del{e) with probability Pr{e). The likelihood [b, a] {w') of a world state w' in the 
belief state [b, a] , resulting from applying a probabilistic action a in b, is given by 



where e is the effect of a such that con{e) C w, and 5{-) is the Kronecker step function that takes 
the value 1 if the argument predicate evaluates to TRUE, and otherwise. 

Our formalism covers all the problem-description features supported by the previously proposed 
formalisms for conformant probabilistic planning (Kushmerick et al., 1995; Majercik & Littman, 
1998; Hyafil & Bacchus, 2004; Onder et al., 2006; Bryce et al, 2006; Huang, 2006), and it cor- 
responds to what is called Unary Nondeterminism (IND) normal form (Rintanen, 2003). We note 
that there are more succinct forms for specifying probabilistic planning problems (Rintanen, 2003), 
yet IND normal form appears to be most intuitive from the perspective of knowledge engineering. 

Example 1 Say we have a robot and a block that physically can be at one of two locations. This 
information is captured by the propositions r\ , r2 for the robot, and b\ , 62 for the block, respec- 
tively. The robot can either move from one location to another, or do it while carrying the block. 
If the robot moves without the block, then its move is guaranteed to succeed. This provides us 
with a pair of symmetrically defined deterministic actions {move-right, move-left}. The ac- 
tion move-right has an empty precondition, and a single conditional effect e = ({ri}, {e}) with 
Pr{£) = 1, add{e) = {r2}, and del{e) = {ri}. If the robot tries to move while carrying the block, 
then this move succeeds with probability 0.7, while with probability 0.2 the robot ends up moving 
without the block, and with probability 0.1 this move of the robot fails completely. This provides us 
with a pair of (again, symmetrically defined) probabilistic actions {move-b-right, move-b-left}. 
The action move-b-right has an empty precondition, and two conditional effects specified as in 



Having specified the semantics and structure of all the components of {A. bj,G,9) but 9, we are 
now ready to specify the actual task of probabilistic planning in our setting. Recall that our actions 
transform probabihstic belief states to belief states. For any action sequence a & A*, and any behef 




b{w) Pr{£)-5{ 

£GA(e) 



w' = w\J s\s' ,s <^ add{e),s' C del{£)) , (1) 



Table 1. 



568 



Probabilistic-FF 



E{a) 


con{e) 


A(e) 


Pr{s) 


add{e) 


del{£) 


e 


ri A bi 


£1 


0.7 


{^2,62} 


{riM} 


£2 


0.2 




in] 


£3 


0.1 








e' 
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Table 1 : Possible effects and outcomes of the action move-b-right in Example 1 . 



state b, the new belief state [b, a] resulting from applying a at 5 is given by 
[b,a] = < 



b, a={) 
[b,a] , a = {a),a £ A 

^ [[b, a] ,a'] , a = (a) - a' ,a & A, a' 7^ 



(2) 



In such setting, achieving G with certainty is typically unreaUstic. Hence, 9 specifies the required 
lower bound on the probability of achieving G. A sequence of actions a is called a plan if we have 
ba{G) > for the beUef state ba = \bi,a\. 



2.2 Specifying the Initial Belief State 

Considering the initial belief state, practical considerations force us to Umit our attention only to 
compactly representable probability distributions bj. While there are numerous alternatives for 
compact representation of structured probability distributions, Bayes networks (BNs) (Pearl, 1988) 
to date is by far the most popular such representation model. ^ Therefore, in Probabilistic-FF we 
assume that the initial belief state bj is described by a BN Mbj over our set of propositions V. 

As excellent introductions to BNs abound (e.g., see Jensen, 1996), it suffices here to briefly 
define our notation. A BN M = {Q, T) represents a probability distribution as a directed acyclic 
graph Q, where its set of nodes X stands for random variables (assumed discrete in this paper), 
and T, a set of tables of conditional probabilities (CPTs) — one table Tx for each node X e X. 
For each possible value x € Dom{X) (where Dom{X) denotes the domain of X), the table Tx 
lists the probability of the event X = x given each possible value assignment to all of its immediate 
ancestors (parents) Pa{X) in Q. Thus, the table size is exponential in the in-degree of X. Usually, it 
is assumed either that this in-degree is small (Pearl, 1988), or that the probabiUstic dependence of X 
on Pa{X) induces a significant local structure allowing a compact representation of Tx (Shimony, 
1993, 1995; Boutilier, Friedman, Goldszmidt, & Koller, 1996). (Otherwise, representation of the 
distribution as a BN would not be a good idea in the first place.) The joint probability of a complete 
assignment "d to the variables X is given by the product of terms taken from the respective 
CPTs (Pearl, 1988): 

Pr{^) = n Pr{d[X] I d[Pa{X)]) = \{Tx {^[X] \ d[Pa{X)]) , 

where ■;?[•] stands for the partial assignment provided by d to the corresponding subset of X. 

2. While BNs are our choice here, our framework can support other models as well, e.g. stochastic decision trees. 
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In Probabilistic-FF we allow to be described over the multi-valued variables underlying 
the planning problem. This significantly simplifies the process of specifying A4j since the STRIPS 
propositions V do not correspond to the true random variables underlying problem specification.^ 
Specifically, let U!Li ^« be a partition of V such that each proposition set Vi uniquely corresponds 
to the domain of a multi-valued variable underlying our problem. That is, for every world state w 
and every Vi, if \Vi\ > 1, then there is exactly one proposition q ^ Vi that holds in w. The variables 
of the BN Mbj describing our initial belief state 6/ are X = {Xi, . . . , X^}, where Dom{Xi) = Vi 
if \Vi\ > 1, and Dom{Xi) = {q, -.q} ifVi = {q}. 

Example 2 For an illustration of such J\fbj , consider our running example, and say the robot is 
known to be initially at one of the two possible locations with probability Pr(ri) = 0.9 and 
Pr{r2) = 0.1. Suppose there is a correlation in our belief about the initial locations of the robot and 
the block. We believe that, if the robot is at ri, then Pr{hi) = 0.7 (and Pr{b2) = 0.3), while if the 
robot is at r2, then Pr{bi) = 0.2 (and Pr{b2) = 0.8). The initial belief state BN Mbj is then defined 
over two variables i? ("robot") and ("block") with L'om(i?) = {ri,r2} wad Dom{B) = {61,62}, 
respectively, and it is depicted in Figure 1. 



n 


r2 


0.9 


0.1 



R 



B 





61 62 


n 


0.7 0.3 


1-2 


0.2 0.8 



Figure 1 : Bayes network A4j for Example 1 . 



It is not hard to see that our STRIPS-style actions a ^ A can be equivalently specified in terms 
of the multi-valued variables X. Specifically, if \Vi\ > 1, then no action a can add a proposition 
q E Vi without deleting some other proposition q' G Vi, and thus, we can consider a as setting 
Xi = q. If \Vi\ = 1, then adding and deleting q EVi has the standard semantics of setting Xi = q 
and Xi = -ig, respectively. For simplicity of presentation, we assume that our actions are not self- 
contradictory at the level of Pd as well — if two conditional effects e, e' G E{a) can possibly occur in 
some world state w, then the subsets of X affected by these two effects have to be disjoint. Finally, 
our goal G directly corresponds to a partial assignment to X (unless our G is self-contradictory, 
requiring q Aq' for some q,q' eVi.) 



3. Belief States 

In this section, we explain our representation of, and reasoning about, belief states. We first explain 
how probabilistic belief states are represented as time-stamped BNs, then we explain how those 
BNs are encoded and reasoned about in the form of weighted CNF formulas. This representation 
of beUef states by weighted CNFs is then illustrated on the belief state from our running example in 
Figure 2. We finally provide the details about how this works in Probabihstic-FF. 

3. Specifying A/i,j directly over V would require identifying the multi-valued variables anyway, followed by connecting 
all the propositions corresponding to a multi- valued variable by a complete DAG, and then normalizing the CPTs of 
these propositions in a certain manner. 
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Figure 2: Bayes network Mb^ for our running Example 1-2 and action sequence 

a = {move-b-right, move-left). 



3.1 Bayesian Networks 

Probabilistic-FF performs a forward search in a space of belief states. The search states are belief 
states (that is, probability distributions over the world states w), and the search is restricted to beUef 
states reachable from the initial belief state bj through some sequences of actions a. A key decision 
one should make is the actual representation of the belief states. Let 6/ be our initial belief state 
captured by the BN Afh^, and let bahe a belief state resulting from applying to bj a sequence of 
actions a. One of the well-known problems in the area of decision-theoretic planning is that the 
description of ba directly over the state variables X becomes less and less structured as the number 
of (especially stochastic) actions in a increases. To overcome this limitation, we represent belief 
states 6a as a BN that explicitly captures the sequential application of a starting from bj, trading 
the representation size for the cost of inference, compared to representing belief states directly as 
distributions over world states. Below we formally specify the structure of such a BN A/^j^, assuming 
that all the actions a are applicable in the corresponding belief states of their application, and later 
showing that Probabilistic-FF makes sure this is indeed the case. We note that these belief-state 
BNs are similar in spirit and structure to those proposed in the AT literature for verifying that a 
probabilistic plan achieves its goals with a certain probability (Dean & Kanazawa, 1989; Hanks & 
McDermott, 1994; Kushmerick et al., 1995). 

Figure 2 illustrates the construction of Mb^ for our running example with a = {move-b-right, 
move-left). In general, let a = (a^ , . . . , a"*) be a sequence of actions, numbered according to their 
appearance on a. For < t < m, let be a repUca of our state variables X, with Xf^-^ e X^t) 
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corresponding to X G X. The variable set of is the union of -^(0), • • • ,^{m)^ plus some 
additional variables that we introduce for the actions in a. 

First, for each X(o) ^ '^(o)' we set the parents Pa(X(o)) and conditional probability tables 
TxjQ) to simply copy these of the state variable X in A/&^ . Now, consider an action a* from a, and 
let a* = a. For each such action we introduce a discrete variable Y(^f-^ that "mediates" between the 
variable layers ^(t-i) -^(t)- The domain of Y^j) is set to Dom{Y(^f^) = Uce-B(a) ^(^)' ^^^^ i^' ^'^ 
the union of probabihstic outcomes of all possible effects of a. The parents of Y(j) in A/&g- are set to 

Pa(Y(,)) = U I con(e) n ZPom(X) 7^ 0} , (3) 

eeE{a) 

and, for each tt G Z)om(Pa(Y'(t))), we set 

rr fv .\^\ jPr{£), con (e(e))C7r 
rn,(%=^k) = |^^ otherw.se ' 

where e(e) denotes the effect e of a such that e G A(e). 

We refer to the set of all such variables Y(t) created for the actions of a as y. Now, let Ex {a) C 
E{a) be the probabilistic effects of a that affect a variable X G Af. If Ex (a) = 0, then we set 
Pa(X(,)) = {X(,_i)},and 

Otherwise, if Ex (a) 7^ 0, let G Dom(X) be the value provided to X by e G A(e), e G Ex{a). 
Recall that the outcomes of effects E{a) are all mutually exclusive. Hence, we set Pa(X(j)) = 
{X^t-i),Y(t-i)}, and 

{1, e(e) G Ex{a) A x = Xg, 
1, e{e) ^Exia) A x = x', , (6) 
0, otherwise 

where e(e) denotes the effect responsible for the outcome e. 

It is not hard to verify that Equations 4-6 capture the frame axioms and probabilistic seman- 
tics of our actions. In principle, this accomphshes our construction of A/^g- over the variables 
Xjy- = y Ut^Lo '^(t) ■ ™te, however, that the mediating variable Y^^j.^ are really needed only 
for truly probabilistic actions. Specifically, if a* is a deterministic action a, let Ex{a) C E{a) be 
the conditional effects of a that add and/or delete propositions associated with the domain of a vari- 
able X G A". If Ex (a) = 0, then we set Pa(X(j)) = and Tx(t) according to Equation 5. 
Otherwise, we set 

Pa(X(,)) = U I con(e) n Dom{X) ^ 0} , (7) 

eeEx(a) 

and specify Tx^^^ as follows. Let Xg G Dom{X) be the value that (the only deterministic outcome 
of) the effect e G Ex{a) provides to X. For each tt G Dom{Pa{X(^^^)), if there exists e G Ex{a) 
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such that con{e) C tt, then we set 



Otherwise, we set 



^ ^ ^ i 0, otherwise 



^(^^^''(^^^"l^^^b, otherwise 



Due to the self-consistency of the action, it is not hard to verify that Equations 8-9 are consistent, 
and, together with Equation 5, capture the semantics of the conditional deterministic actions. This 
special treatment of deterministic actions is illustrated in Figure 2 by the direct dependencies of 

Proposition 1 Let (A, M^i ,G,6) bea probabilistic planning problem, and a be an m-step sequence 
of actions applicable in hj. Let Pr be the joint probability distribution induced by Aff^ on its 
variables Xf^. The belief state ha corresponds to the marginal distribution of Pr on X^^y that is: 
ba{X) = Pr{X(^^-^), and ifG(^^-^ is a partial assignment provided by G to A'^^^, then the probability 
ha{G) that a achieves G starting from hj is equal to Pr(G(m))- 

As we already mentioned, our belief-state BNs are constructed along the principles outlined 
and used by Dean and Kanazawa (1989), Hanks and McDermott (1994), and Kushmerick et al. 
(1995), and thus the correctness of Proposition 1 is immediate from these previous results. At this 
point, it is worth bringing attention to the fact that all the variables in , . . . , ^(m) are completely 
deterministic. Moreover, the CPTs of all the variables of M},^ are all compactly representable due to 
either a low number of parents, or some local structure induced by a large amount of context-specific 
independence, or both. This compactness of the CPTs in Af^^ is implied by the compactness of the 
STRlPS-style specification of the planning actions. By exploiting this compactness of the action 
specification, the size of the A^j^-'s description can be kept linear in the size of the input and the 
number of actions in a. 



Proposition 2 Let {A,Mbj, G, 9) be a probabilistic planning problem described over k state vari- 
ables, and a be an m-step sequence of actions from A. Then,wehave \Mb^\ = 0{\Afbj\+ma{k+l)) 
where a is the largest description size of an action in A. 

The proof of Proposition 2, as well as the proofs of other formal claims in the paper, are relegated 
to Appendix A, pp. 613. 



3.2 Weighted CNFs 

Given the representation of belief states as BNs, next we should select a mechanism for reasoning 
about these BNs. In general, computing the probability of a query in BNs is known to be #P- 
complete (Roth, 1996). In addition, it is not hard to verify, using an analysis similar to the ones 
of Darwiche (2001) and Brafman and Domshlak (2006), that the networks arising in our work 
will typically exhibit large tree-width. While numerous exact algorithms for inference with BNs 
have been proposed in the literature (Darwiche, 2000; Dechter, 1999; Zhang & Poole, 1994), the 
classical algorithms do not scale well on large networks exhibiting high tree- width. On the positive 
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side, however, an observation that guides some recent advances in the area of probabihstic reasoning 
is that real-world domains typically exhibit a significant degree of deterministic dependencies and 
context-specific independencies between the problem variables. Targeting this property of practical 
BNs already resulted in powerful inference techniques (Chavira & Darwiche, 2005; Sang et al., 
2005). The general principle underlying these techniques is to 

(i) Compile a BN M into a weighted propositional logic formula (t>{M) in CNF, and 

(ii) Perform an efficient weighted model counting for ^(AA) by reusing (and adapting) certain 
techniques that appear powerful in enhancing backtracking DPLL-style search for SAT. 

One observation we had at the early stages of developing Probabilistic-FF is that the type of 
networks and type of queries we have in our problems make this machinery for solving BNs by 
weighted CNF model counting very attractive for our needs. First, in Section 3.1 we have already 
shown that the BNs representing our belief states exhibit a large amount of both deterministic nodes 
and context-specific independence. Second, the queries of our interest correspond to computing 
probabiUty of the "evidence" G(m) in Mi^, and this type of query has a clear interpretation in terms 
of model counting (Sang et al., 2005). Hence, taking this route in Probabilistic-FF, we compile our 
belief state BNs to weighted CNFs following the encoding scheme proposed by Sang et al. (2005), 
and answer probabilistic queries using Cachet (Sang, Bacchus, Beame, Kautz, & Pitassi, 2004), one 
of the most powerful systems to date for exact weighted model counting in CNFs. 

In general, the weighted CNFs and the weights of such formulas are specified as follows. Let 
V = {Vi, . . . , V„} be a set of propositional variables with Dom{Vi) = {vi, -^vi], and let : 
(Jj DomiVi) be a non-negative, real-valued weight function from the literals of V. For any 

partial assignment tt to V, the weight ro(7r) of this assignment is defined as the product of its literals' 
weights, that is, tx7(7r) = Wi^^ w{l). Finally, a propositional logic formula (j) is called weighted if it 
is defined over such a weighted set of propositional variables. For any weighted formula over V, 
the weight tu(^) is defined as the sum of the weights of all the complete assignments to V satisfying 
0, that is, 

^(^) = XI t^(7r)(5(7r ^ 

7re£)om(V) 

where Dom{V) = XiDom{Vi). For instance, if for all variables Vi we have zulvi) = w{-'Vi) = 1, 
then ti7(0) simply stands for the number of complete assignments to V that satisfy 0. 

Given an initial belief state BN Aft^, and a sequence of actions a = (a^, . . . , a"*) applicable in 
bi, here we describe how the weighted CNF encoding (f){Afb^) (or (f){ba), for short) of the belief state 
ba is built and used in Probabilistic-FF. First, we formally specify the generic scheme introduced 
by Sang et al. (2005) for encoding a BN J\f over variables X into a weighted CNF (t>{J\f). The 
encoding formula 0(A/') contains two sets of variables. First, for each variable Z ^ X and each 
value z G Dom{Z), the formula (f){J\f) contains a state proposition with literals {z, -iz}, weighted 
as zu{z) = w{-^z) = 1. These state propositions act in (/)(6a) as regular SAT propositions. Now, 
for each variable Z € Xh^, let Dom{Z) = {zi, . . . , z^} be an arbitrary fixed ordering of Dom{Z). 
Recall that each row Tz [i\ in the CPT of Z corresponds to an assignment C,i (or a set of such as- 
signments) to Pa{Z). Thus, the number of rows in Tz is upper bounded by the number of different 
assignments to Pa{Z), but (as it happens in our case) it can be significantly lower if the depen- 
dence of Z on Pa{Z) induces a substantial local structure. Following the ordering of Dom{Z) as 
above, the entry Tz{i,j] contains the conditional probability of Pr{zj \ Ci)- For every CPT entry 
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procedure basic-WMC(^) 
if </) = return 1 
if (f) has an empty clause return 
select a variable V & cj) 

return basic-V\/MC((;/!)|t,) • w{v) + basic-WMC((;/!)|^^) ■ w{-^v) 

Figure 3: Basic DPPL-style weighted model counting. 

Tz [i, j] but the last one (i.e., Tz [i, k]), the formula 4>{J\f) contains a chance proposition with literals 
{(z* ), These chance variables aim at capturing the probabilistic information from the CPTs 

of Ni^. Specifically, the weight of the literal (2;*) is set to Pr{zj \ Q, -izi, . . . , ^zj-i), that is to 
conditional probability that the entry is true, given that the row is true, and no prior entry in the row 
is true: 

- m = — — 

^H4)) = i-^((4)) 

Considering the clauses of (t>{M), for each variable Z ^ X, and each CPT entry Tz[i,j], the 
formula <p{J\f) contains a clause 

{CiA^{z\)A---A^{z}_,)A{z}))^zj, (11) 

where Q is a conjunction of the literals forming the assignment Q € Dom{Pa{Z)). These clauses 
ensure that the weights of the complete assignments to the variables of (t){N) are equal to the prob- 
ability of the corresponding atomic events as postulated by the BN M. To illustrate the construction 
in Equations 10-11, let boolean variables A and B be the parents of a ternary variable C (with 
Dom{C) = {Ci, C2, C3}) in some BN, and let Pr{Ci\A, ^B) = 0.2, Pr{C2\A, -^B) = 0.4, and 
Pr{C2,\A, -iS) = 0.4. Let the raw corresponding to the assignment A, ^B to Pa{C) be the i-th 
row of the CPT Tq. In the encoding of this BN, the first two entries of this raw of Tc are captured 
by a pair of respective chance propositions (CJ), and (C2). According to Equation 10, the weights 
of these propositions are set to w {{CD) = 0.2, and w {{CD) = = 0.5. Then, according to 

Equation 11, the encoding contains three clauses 

V5 V-(Cj) VCi) 

(-^ v5 V (Ci) v-(c^) VC2) 

{-nAwBW {CD V {C^yCa) 

Finally, for each variable Z ^ X, the formula contains a standard set of clauses encoding 
the "exactly one" relationship between the state propositions capturing the value of Z. This accom- 
plishes the encoding of M into (t>{M). In the next Section 3.3 we illustrate this encoding on the 
belief state BN from our running example. 

The weighted CNF encoding (t){ha) of the belief state BN Mb^ provides the input to a weighted 
model counting procedure. A simple recursive DPPL-style procedure basic-WMC underlying Ca- 
chet (Sang et al., 2004) is depicted in Figure 3, where the formula is obtained from by setting 
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the literal v to true. Theorem 3 by Sang et al. (2005) shows that if is a weighted CNF encoding 
of a BN M, and Pr{Q\E) is a general query with respect to J\f, query Q, and evidence E, then we 
have: 

_ basic-WMC(</,AQAE) 
PnQ\E)- basic-WMC(0AS) ' ^^^^ 

where query Q and evidence can in fact be arbitrary formulas in propositional logic. Note that, 
in a special (and very relevant to us) case of empty evidence, Equation 12 reduces to Pr(Q) = 
basic-WMC(0AQ), that is, a single call to the basic-WMC procedure. Corollary 3 is then immediate 
from our Proposition 1 and Theorem 3 by Sang et al. (2005). 

Corollary 3 Let {A, bi, G, 0) be a probabilistic planning task with a BN Mbj describing hi, and a 
be an m-step sequence of actions applicable in hj. The probability ha{G) that a achieves G starting 
from hi is given by: 

ha{G) = WMC (<^(M A G(m)) , (13) 
where G{m) is a conjunction of the goal literals time-stamped with the time endpoint m of a. 

3.3 Example: Weighted CNF Encoding of Belief States 

We now illustrate the generic BN-to-WCNF encoding scheme of Sang et al. (2005) on the belief 
state BN Mh^ from our running example in Figure 2. 

ForO <i <2, we introduce time-stamped state propositions ri{i)^r2{i)-,bi{i),b-2{i). Likewise, 
we introduce four state propositions £2(1), £3(1), e'i(l) corresponding to the values of the 

variable Y^iy The first set of clauses in ^(6a) ensure the "exactly one" relationship between the 
state propositions capturing the value of a variable in A/'&jj.: 



(ei(l) V£2(l) V£3(l) V4(l)) , 
1 < i < j < 4 : 

h2/.(l)V-%(l)), 



(14) 

< i < 2 : 



(ri(i) V r2(i)) , (^ri(i) V -.r2(i)) 
V62«), V-52(i)) 



Now we proceed with encoding the CPTs of Nh^. The root nodes have only one row in their 
CPTs so their chance propositions can be identified with the corresponding state variables (Sang 
et al., 2005). Hence, for the root variable i?(o) we need neither additional clauses nor special 
chance propositions, but the state proposition ri(0) of (l){ha) is treated as a chance proposition 
withtu(ri(0)) = 0.9. 

Encoding of the variable B^q) ^ bit more involved. The CPT Tb(o) contains two (content-wise 
different) rows corresponding to the "given r\" and "given r2" cases, and both these cases induce 
a non-deterministic dependence of S(o) on i?(o)- To encode the content of ?B(o) we introduce 
two chance variables (61 (0)^) and (61(0)^) with w{{hi{fdf)) = 0.7 and w{{hi{{)f)) = 0.2. The 
positive literals of (&i(0)^) and (61 (0)^) capture the events "61 given ri" and "5i given r2", while 
the negations -1(61 (0)^) and -1(61 (0)^) capture the complementary events "62 given r2" and "62 
given r2", respectively. Now consider the "given ri" row in ^5^^^ . To encode this row, we need 



576 



Probabilistic-FF 



(j){ba) to contain (ri(0) A (61 (0)^)) 61 (0) and (ri(0) A -(61 (0)^)) 62(0). Similar encoding 
is required for the row "given r2", and thus the encoding of Tgo introduces four additional clauses: 

(^ri(O) V -(61 (0)1) V 61 (0)) , (^ri(O) V (61 (0)^) V 62(0)) 
(-r2(0) V -(61 (0)2) V 61 (0)) , (-r2(0) V (61 (0)^) V 62(0)) 

Having finished with the J\fbj part of Mi^, we proceed with encoding the variable Y(i) cor- 
responding to the probabilistic action move-b-right. To encode the first row of Ty^^^ we in- 
troduce three chance propositions (ei(l)i), (£2(1)^), and (£3(1)^); in general, no chance vari- 
ables are needed for the last entries of the CPT rows. The weights of these chance propositions 
are set according to Equation 10 to w ((ei(l)i)) = 0.7, w ((£2(1)^)) = x=F7 = 0.6(6), and 
w ((£3(1)^)) = jrrg-g = O-l- Using these chance propositions, we add to ^(6a) four clauses as in 
Equation 1 1, notably the first four clauses of Equation 16 below. 

Proceeding the second row of Ty^^j, observe that the value of i?(o) and i?(o) in this case fully 
determines the value of Y(^iy This deterministic dependence can be encoded without using any 
chance propositions using the last two clauses in Equation 16. 

(-ri(O) V -61(0) V -(£i(l)^) V £i(l)) , 

(-ri(O) V -61(0) V (£i(l)i) V -(£2(1)') V £2(1)) , 

(^n(o) V ^61 (0) V (£i(i)i) V (£2(1)1) V ^(,3(1)1) V £3(1)) , 

(-ri(O) V -61 (0) V (£i(l)i) V (£2(1)1) V (^3(1)1) V e[il)) , 



(ri(O)vVi(l)), (6i(0)vVi(l)) 

Using the state/chance variables introduced for R'^, B^, and Y'(i), we encode the CPTs of 
and 5(1) as: 

i?(i) :(-£i(l)Vr2(l)), (-£2(l)Vr2(l)), 

(-£3(1) V -ri(0) V ri(l)) , (Vi(l) V -ri(0) V ri(l)) , 
(-£3(1) V -ri(0) V ri(l)) , (-£'i(l) V -r2(0) V r2(l)) 

5(1) :(-£i(l)V 62(1)), 

(£i(l)V -61(0) V 61(1)), 
(£i(l)V -62(0) V 62(1)) 

Since the CPTs of both i?(i) and 5(i) are completely deterministic, their encoding as well is using 
no chance propositions. Finally, we encode the (deterministic) CPTs of i2(2) and 5(2) as: 

R(2):{ri{2)) 

5(2) : (-61(1) V 61 (2)) (18) 

(-62(1) V 62(2)) 

where the unary clause (^1(2)) is a reduction of (-iri(l) V 7-1(2)) and (-ir2(l) V ri(2)). This ac- 
compUshes our encoding of 0(6a)- 
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3.4 From Conformant-FF to Probabilistic-FF 

Besides the fact that weighted model counting is attractive for the kinds of BNs arising in our con- 
text, the weighted CNF representation of belief states works extremely well with the ideas underly- 
ing Conformant-FF (Hoffmann & Brafman, 2006). This was outhned in the introduction already; 
here we give a few more details. 

As stated, Conformant-FF does a forward search in a non-probabiUstic beUef space in which 
each belief state corresponds to a set of world states considered to be possible. The main trick of 
Conformant-FF is the use of CNF formulas for an implicit representation of belief states, where 
formulas (j){a) encode the semantics of executing action sequence a in the initial behef state. Facts 
known to be true or false are inferred from these formulas. This computation of only a partial 
knowledge constitutes a lazy kind of belief state representation, in comparison to other approaches 
that use explicit enumeration (Bonet & Geffner, 2000) or BDDs (Bertoli, Cimatti, Pistore, Roveri, 
& Traverso, 2001) to fully represent belief states. The basic ideas underlying Probabilistic-FF are: 

(i) Define time-stamped Bayesian Networks (BN) describing probabilistic belief states (Sec- 
tion 3.1 above). 

(ii) Extend Conformant-FF's behef state CNFs to model these BN (Section 3.2 above). 

(iii) In addition to the SAT reasoning used by Conformant-FF, use weighted model-counting to 
determine whether the probability of the (unknown) goals in a belief state is high enough 
(directly below). 

(iv) Introduce approximate probabihstic reasoning into Conformant-FF's heuristic function (Sec- 
tion 4 below). 

In more detail, given a probabilistic planning task {A, bj, G, 6), a belief state ha corresponding to 
some applicable in 6/ m-step action sequence a, and a proposition q e V, we say that q is known 
in ba if ba{q) = 1, negatively known in ba if ba{q) = 0, and unknown in ba, otherwise. We begin 
with determining whether each q is known, negatively known, or unknown at time m. Re-using the 
Conformant-FF machinery, this classification requires up to two SAT tests of (/){ba) A ->q{m) and 
4>{ba) A q{m), respectively. The information provided by this classification is used threefold. First, 
if a subgoal ^ G G is negatively known at time m, then we have ha{G) = 0. On the other extreme, 
if all the subgoals of G are known at time m, then we have ba{G) = 1. Finally, if some subgoals of 
G are known and the rest are unknown at time m, then we accomplish evaluating the behef state ba 
by testing whether 

ba{G) = WMC ((^(M A G{m)) > 9. (19) 

Note also that having the sets of all (positively /negatively) known propositions at all time steps up 
to m allows us to significantly simplify the CNF formula 0(60) ^ G{m) by inserting into it the 
corresponding values of known propositions. 

After evaluating the considered action sequence a, if we get ba{G) > 9, then we are done. 
Otherwise, the forward search continues, and the actions that are applicable in ba (and thus used to 
generate the successor belief states) are actions whose preconditions are all known in ba- 
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4. Heuristic Function 

The key component of any heuristic search procedure is the heuristic function. The quahty (in- 

formedness) and computational cost of that function determine the performance of the search. The 
heuristic function is usually obtained from solutions to a relaxation of the actual problem of in- 
terest (Pearl, 1984; Russell & Norvig, 2004). In classical planning, a successful idea has been to 
use a relaxation that ignores the delete effects of the actions (McDermott, 1999; Bonet & Geffner, 
2001; Hoffmann & Nebel, 2001). In particular, the heuristic of the FF planning system is based on 
the notion of relaxed plan, which is a plan that achieves the goals while assuming that all delete 
lists of actions are empty. The relaxed plan is computed using a Graphplan-style (Blum & Furst, 
1997) technique combining a forward chaining graph construction phase with a backward chaining 
plan extraction phase. The heuristic value h{w) that FF provides to a world state w encountered 
during the search is the length of the relaxed plan from w. In Conformant-FF, this methodology was 
extended to the setting of conformant planning under initial state uncertainty (without uncertainty 
about action effects). Herein, we extend Conformant-FF's machinery to handle probabilistic initial 
states and effects. Section 4. 1 provides background on the techniques used in FF and Conformant- 
FF, then Sections 4.2 and 4.4 detail our algorithms for the forward and backward chaining phases in 
Probabilistic-FF, respectively. These algorithms for the two phases of the Probabilistic-FF heuristic 
computation are illustrated on our running example in Sections 4.3 and 4.5, respectively. 

4.1 FF and Conformant-FF 

We specify how relaxed plans are computed in FF; we provide a coarse sketch of how they are 
computed in Conformant-FF. The purpose of the latter is only to slowly prepare the reader for what 
is to come: Conformant-FF's techniques are re-used for Probabilistic-FF anyway, and hence will be 
described in full detail as part of Sections 4.2 and 4.4. 

Formally, relaxed plans in classical planning are computed as follows. Starting from w, FF 
builds a relaxed planning graph as a sequence of alternating proposition layers P{t) and action 
layers A{t), where -P(O) is the same as w, A{t) is the set of all actions whose preconditions are 
contained in P{t), and P{t + 1) is obtained from P{t) by including the add effects (with fulfilled 
conditions) of the actions in A{t). That is, P{t) always contains those facts that will be true if one 
would execute (the relaxed versions of) all actions at the earlier layers up to A(t — 1). The relaxed 
planning graph is constructed either until it reaches a propositional layer P{m) that contains all 
the goals, or until the construction reaches a fixpoint P(t) = P(t + \) without reaching the goals. 
The latter case corresponds to (all) situations in which a relaxed plan does not exist, and because 
existence of a relaxed plan is a necessary condition for the existence of a real plan, the state w is 
excluded from the search space by setting h{w) = oo. In the former case of G C P{m), a relaxed 
plan is a subset of actions in ^(1), ... , A{m) that suffices to achieve the goals (under ignoring the 
delete lists), and it can be extracted by a simple backchaining loop: For each goal in P{m), select 
an action in ^(1), ... , A{'m) that achieves this goal, and iterate the process by considering those 
actions' preconditions and the respective effect conditions as new subgoals. The heuristic estimate 
h{w) is then set to the length of the extracted relaxed plan, that is, to the number of actions selected 
in this backchaining process. 

Aiming at extending the machinery of FF to conformant planning, in Conformant-FF, Hoff- 
mann and Brafman (2006) suggested to extend the relaxed planning graph with additional fact lay- 
ers uP{t) containing the facts unknown at time t, and then to reason about when such unknown 
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facts become known in the relaxed planning graph. As the complexity of this type of reasoning is 
prohibitive, Conformant-FF further relaxes the planning task by ignoring not only the delete lists, 
but also all but one of the unknown conditions of each action effect. That is, if action a appears 
in layer A{t), and for effect e of a we have con(e) C P(t) U uP{t) and con(e) fl uP{t) / 0, 
then con{e) fl uP{t) is arbitrarily reduced to contain exactly one literal, and reasoning is done as if 
con{e) had this reduced form from the beginning. 

This relaxation converts imphcations {/\cecon{e)nuP{t) "^(0) ~^ + 1) ^^^^ action effects 
induce between unknown propositions into their 2-projections that take the form of binary impli- 
cations c{t) — )■ q{t + 1), for arbitrary c G con(e) n uP{t). Due to the layered structure of the 
planning graph, the set of all these binary implications c(t) — > q{t + 1) can be seen as forming a 
directed acychc graph Imp. Under the given relaxations, this graph captures exactly all dependen- 
cies between the truth of propositions over time. Hence, checking whether a proposition q becomes 
known at time t can be done as follows. First, backchain over the implication edges of Imp that end 
in q{t), and collect the set support{q{t)) of leafs^ that are reached. Then, if $ is the CNF formula 
describing the possible initial states, test by a SAT check whether 



This test will succeed if and only if at least one of the leafs in support{q{t)) is true in every possible 
initial state. Under the given relaxations, this is the case if and only if, when applying all actions in 
the relaxed planning graph, q will always be true at time t.^ 

The process of extracting a relaxed plan from the constructed conformant relaxed planning 
graph is an extension of FF's respective process with machinery that selects actions responsible for 
relevant paths in Imp. The overall Conformant-FF heuristic machinery is sound and complete for 
relaxed tasks, and yields a heuristic function that is highly informative across a range of challenging 
domains (Hoffmann & Brafman, 2006). 

In this work, we adopt Conformant-FF's relaxations, ignoring the delete lists of the action ef- 
fects, as well as all but one of the propositions in the effect's condition. Accordingly, we adopt the 
following notations from Conformant-FF. Given a set of actions A, we denote by \'l any function 
from A into the set of all possible actions, such that maps each a G ^ to the action similar to a 
but with empty delete lists and with all but one conditioning propositions of each effect removed; 
for \i{a), we write a\f . By A\f we denote the action set obtained by applying l^'" to all the actions 
of A, that is, yl|]'" = {a|]^jaG74}. For an action sequence a we denote by a\i the sequence of 
actions obtained by applying |^ to every action along a, that is. 



For a probabilistic planning task {A,bj,G,9), the task {A\f ,bj,G,6) is called a relaxation of 
{A,bj,G,d). Finally, if is a plan for {A\f ,bi,G,0), then a is called a relaxed plan for 



4. Following the Conformant-FF terminology, by "leafs" we refer to the nodes having zero in-degree. 

5. Note here that it would be possible to do a full SAT check, without any 2-projection (without relying on Imp), to see 

whether q becomes known at t. However, as indicated above, doing such a full check for every unknown proposition 
at every level of the relaxed planning graph for every search state would very likely be too expensive, computationally. 



V ' 



lesuppoTt{q{t)) 




{A,bi,G,e). 
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In the next two sections we describe the machinery underlying the ProbabiUstic-FF heuristic 
estimation. Due to the similarity between the conceptual relaxations used in Probabilistic-FF and 
Conformant-FF, Probabilistic-FF inherits almost all of Conformant-FF's machinery. Of course, 
the new contributions are those algorithms dealing with probabilistic belief states and probabilistic 
actions. 

4.2 Probabilistic Relaxed Planning Graphs 

Like FF and Conformant-FF, Probabilistic-FF computes its heuristic function in two steps, the first 
one chaining forward to build a relaxed planning graph, and the second step chaining backward to 
extract a relaxed plan. In this section, we describe in detail Probabilistic-FF' s forward chaining step, 
building a probabilistic relaxed planning graph (or PRPG, for short). In Section 4.4, we then show 
how one can extract a (probabilistic) relaxed plan from the PRPG. We provide a detailed illustration 
of the PRPG construction process on the basis of our running example; since the illustration is 
lengthy, it is moved to a separate Section 4.3. 

The algorithms building a PRPG are quite involved; it is instructive to first consider (some 
of) the key points before delving into the details. The main issue is, of course, that we need to 
extend Conformant-FF's machinery with the ability to determine when the goal set is sufficiently 
likely, rather than when it is known to be true for sure. To achieve that, we must introduce into 
relaxed planning some effective reasoning about both the probabilistic initial state, and the effects 
of probabilistic actions. It turns out that such a reasoning can be obtained by a certain weighted 
extension of the implication graph. In a nutshell, if we want to determine how likely it is that a fact 
q is true at a time t, then we propagate certain weights backwards through the implication graph, 
starting in q{t); the weight of q{t) is set to 1, and the weight for any p{t') gives an estimate of the 
probability of achieving q at t given that p holds at t'. Computing this probability exactly would, 
of course, be too expensive. Our estimation is based on assuming independence of the various 
probabilistic events involved. This is a choice that we made very carefully; we experimented widely 
with various other options before deciding in favor of this technique. 

Any simplifying assumption in the weight propagation constitutes, of course, another relaxation, 
on top of the relaxations we already inherited from Conformant-FF. The particularly problematic 
aspect of assuming independence is that it is not an under-estimating technique. The actual weight 
of a node p{t') - the probability of achieving q att given that p holds at t' - may be lower than our 
estimate. In effect, the PRPG may decide wrongly that a relaxed plan exists: even if we execute 
all relaxed actions contained in the successful PRPG, the probability of achieving the goal by this 
execution may be less than the required threshold. In other words, we lose the soundness (relative 
to relaxed tasks) of the relaxed planning process. 

We experimented with an alternative weight propagation method, based on an opposite assump- 
tion, that the relevant probabilistic events always co-occur, and that hence the weights must be 
propagated according to simple maximization operations. This propagation method yielded very 
uninformative heuristic values, and hence inacceptable empirical behaviour of Probabilistic-FF, 
even in very simple benchmarks. In our view, it seems unlikely that an under-estimating yet in- 
formative and efficient weight computation exists. We further experimented with some alternative 
non under-estimating propagation schemes, in particular one based on assuming that the probabilis- 
tic events are completely disjoint (and hence weights should be added); these schemes gave better 
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performance than maximization, but lagged far behind the independence assumption in the more 
challenging benchmarks. 

Let us now get into the actual algorithm building a PRPG. A coarse outhne of the algorithm is as 
follows. The PRPG is built in a layer-wise fashion, in each iteration extending the PRPG, reaching 
up to time t, by another layer, reaching up to time t + 1. The actions in the new step are those whose 
preconditions are known to hold at t. Effects conditioned on unknown facts (note here the reduction 
of effect conditions to a single fact) constitute new edges in the implication graph. In difference to 
Conformant-FF, we don't obtain a single edge from condition to add effect; instead, we obtain edges 
from the condition to "chance nodes", where each chance node represents a probabiUstic outcome of 
the effect; the chance nodes, in turn, are linked by edges to their respective add effects. The weights 
of the chance nodes are set to the probabilities of the respective outcomes, the weights of all other 
nodes are set to 1. These weights are "static weights" which are not "dynamically" modified by 
weight propagation; rather, the static weights form an input to the propagation. 

Once all implication graph edges are inserted at a layer, the algorithm checks whether any new 
facts become known. This check is done very much hke the corresponding check in Conformant-FF, 
by testing whether the disjunction of the support leafs for a proposition p at t + 1 is implied by the 
initial state formula. The two differences to Conformant-FF are: (1) Only leafs are relevant whose 
dynamic weight is 1 (otherwise, achieving a leaf is not guaranteed to accomplish p at t + 1). (2) 
Another reason for p to become known may be that all outcomes of an unconditional effect (or an 
effect with known condition) result in achievement of p at time t+1. We elegantly formulate the 
overall test by a single implication test over support leafs whose dynamic weight equals their own 
weight. 

Like FF's and Conformant-FF's algorithms, the PRPG process has two termination criteria. The 
PRPG terminates positively if the goal probability is high enough at time t; the PRPG terminates 
negatively if, from tto t + 1, nothing has changed that may result in a higher goal propability at 
some future t'. The goal probability in a layer t is computed based on weighted model counting over 
a formula derived from the support leafs of all goals not known to be true. The criteria for negative 
termination check: whether any new facts have become known or unknown (not negatively known); 
whether any possibly relevant new support leafs have appeared; and whether the goal probability 
has increased. If neither is the case, then we can stop safely — if the PRPG terminates unsuccessfully 
then we have a guarantee that there is no relaxed plan, and that the corresponding belief is hence a 
dead end. 

Let us get into the details. Figure 4 depicts the main routine for building the PRPG for a belief 

state ba- As we already specified, the sets P{t), uP{t), and A{t) contain the propositions that are 
known to hold at time t (hold at t with probability 1), the propositions that are unknown to hold at 
time t (hold at t with probability less than 1 but greater than 0), and actions that are known to be 
applicable at time t, respectively. The layers f > of PRPG capture applying the relaxed actions 
starting from ba- The layers —m to —1 of PRPG correspond to the m-step action sequence a leading 
from the initial belief state to the belief state in question ha- We inherit the latter technique from 
Conformant-FF; in a sense, the PRPG "reasons about the past". This may look confusing at first 
sight, but it has a simple reason. Imagine the PRPG starts at level instead. Then, to check whether 
a proposition becomes known, we have to do SAT tests regarding support leafs against the belief 
state formula, (t>{ba), instead of the initial state formula (similarly for weighted model counting 
to test whether the goal is likely enough). Testing against 0(6a) is possible, but very expensive 
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procedure build-PRPG (a. A, MMhA.G, 9. |+), 

returns a Bool saying if there is a relaxed plan for the belief state 

given by a = (a~™, . . . , a~^), and 
builds data structures from which a relaxed plan can be extracted 

$:= </.(A/-(,J, /mp := 

P{—m) := {p\p is known in $}, uP{—m) := {p\p is unknown in $} 
for t := m 1 do 

^(t) := {a^\+}VJNOOPS 

build-timestep(i, 
endfor 
i := 

whUe get-P(t, G) < 61 do 

^(t) := {a|+ I a G A,pre{a) C P(t)} u NOOPS 

build-timestep(t, 

ifP(t + 1) = and 

MP(t + 1) = uP{t) and 

€ uP{t + 1) : uP{—m) n support{p{t + 1)) = uP{—m) fl support{p{t)) and 

get-P(t + 1, G) = get-P(t, G) then 

return FALSE 
endif 
t:=i + l 
endwhile 

T := t return TRUE 

Figure 4: Main routine for building a probabilistic relaxed planning graph (PRPG). 

computationally.^ The negative-index layers chain the impUcation graph all the way back to the 
initial state, and hence enable us to perform SAT tests against the - typically much smaller - initial 
state formula. 

Returning to Figure 4, the PRPG is initiaUzed with an empty imphcation set Imp, P{—m) 
and uP{—m) are assigned the propositions that are known and unknown in the initial belief state, 
and a weighted CNF formula $ is initiaUzed with ^(A/j,^). $ is the formula against which impUca- 
tion/weighted model checking tests are run when asking whether a proposition becomes known/whether 
the goal is likely enough. While the PRPG is built, ^> is incrementally extended with further clauses 
to capture the behavior of different effect outcomes. 

The for loop builds the sets P and uP for the a's time steps — m • • • — 1 by iterative invocation 
of the build-timestep procedure that each time expands PRPG by a single time level. At each 
iteration —m < t < —1, the sets P{t + 1) and uP{t + 1) are made to contain the propositions 
that are known/unknown after applying the relaxed version of the action a* G a (remember that 
a = (a^, . . . , a^)). To simpUfy the presentation, each action set A{t) contains a set of dummy 
actions NOOPS that simply transport all the propositions from time layer t to time layer More 
formally, NOOPS = {noopp | p G V}, where pre(noopp) = 0, £^(noopp) = {({p}, {e})}, and 
e = (l.O,M,0)}). 

6. In Conformant-FF, this configuration is implemented as an option; it significantly slows down the search in most 
domains, and brings advantages only in a few cases. 
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The subsequent while loop constructs the relaxed planning graph from layer onwards by, 
again, iterative invocation of the build-timestep procedure. The actions in each layer f > are 
relaxations of those actions whose preconditions are known to hold at time t with certainty. This 
iterative construction is controlled by two termination tests. First, if the goal is estimated to hold at 
layer t with probability higher than 9, then we know that a relaxed plan estimate can be extracted. 
Otherwise, if the graph reaches a fix point, then we know that no relaxed (and thus, no real) plan 
from hi exists. We postpone the discussion of these two termination criteria, and now focus on the 
time layer construction procedure build-timestep. 

procedure build-timestep (t, A), 

builds P{t + l), uP{t + 1), and the implication edges from tXot+1, 

as induced by the action set A 
P{t + l) := P{t),uP{t+l) :=0 

for all effects e of an action a & A, con{e) G P(t) U uP{t) do 
for all e e A(e) do 

uP{t + 1) := uP{t + 1) U add{e) 

introduce new fact s{t) with zu{s{t)) = Pr{e) 

Imp:=Imp\j{{£{t),p{t + l)) |pG add{£)} 
endfor 

if con{e) G uP{t) then 

Imp := ImpU |JeeA(e){(cori(e)(t),e(i))} 
else 

$ := $ A (v,eA(e)£(t)) A Ae,e'eA(e) (^^W V -e'W) 
endif 
endfor 

for all p G uP{t+l) do 

build-w-impleafs(]3(f + l),Imp) 

support{p{t + 1)) ■.= {l\l e leafs{Imp^^,(,^^-^) A ti7p((+i)(/) = zu(l)} 
^'^^ ViesMp(t+i)) I then P(t + 1) P{t + 1) U {p} endif 
endfor 

uP{t + 1) := uP{t + 1) \ P{t + 1) 



Figure 5: Building a time step of the PRPG. 

The build-timestep procedure is shown in Figure 5. The first for loop of build-timestep proceeds 
over all outcomes of (relaxed) actions in the given set A that may occur at time t. For each such 
probabilistic outcome we introduce a new chance proposition weighted by the conditional likelihood 
of that outcome.^ Having that, we extend Imp with binary implications from this new chance 
proposition to the add Ust of the outcome. If we are uncertain about the condition con(e) of the 
corresponding effect at time t, that is, we have con(e) G uP{t), then we also add implications 
from con(e) to the chance propositions created for the outcomes of e. Otherwise, if con(e) is 
known at time t, then there is no uncertainty about our ability to make the effect e to hold at time 
t. In this case, we do not "ground" the chance propositions created for the outcomes of e into the 
implication graph, but simply extend the running formula $ with clauses capturing the "exactly 
one" relationship between these chance propositions corresponding to the alternative outcomes of e 

7. Of course, in our implementation we have a special case treatment for deterministic actions, using no chance nodes 
(rather than a single "chance node" with static weight 1). 



584 



Probabilistic-FF 



at time t. This way, the probabiUstic uncertainty about the outcome of e can be treated as if being a 
property of the initial belief state 6/; This is the only type of knowledge we add into the knowledge 

base formula $ after initializing it in build-PRPG to c6(A/f,^). 



Notation 


Description 




The graph containing exactly all the paths from node v to node u in Imp. 




The subgraph of Imp formed by node u and all the ancestors of u in Imp. 


leafs{Imp') 


The set of all zero in-degree nodes in the subgraph Imp' of Imp. 


E{Imp') 


The set of time-stamped action effects responsible for the implication edges 
of the subgraph Imp' of Imp. 



Table 2: Overview of notations around the implication graph. 

The second for loop checks whether a proposition p, unknown at time t, becomes known at 
time t + 1. This part of the build-timestep procedure is somewhat more involved; Table 2 provides 
an overview of the main notations used in the follows when discussing the various uses of the 
implication graph Imp. 

First thing in the second for loop of build-timestep, a call to build-w-impleafs procedure asso- 
ciates each node v{t') in Imp_^p(j_,_]^) with an estimate {v{t')) on the probability of achieving 
p at time t + 1 by the effects E{Imp^(^f,^_^p(^f_^_i^), given that v holds at time t'. In other words, the 
dynamic weight (according to p{t + 1)) of the implication graph nodes is computed. Note that v{t') 
can be either a time-stamped proposition q{t') for some q & V, or a chance proposition e{t') for 
some probabilistic outcome e. 

We will discuss the build-w-impleafs procedure in detail below. For proceeding to understand 
the second for loop of build-timestep, the main thing we need to know is the following lemma: 

Lemma 4 Given a node v{t') G Imp_^p(|+i), we have ti7p(^+i) (^(^O) — ("^(^0) if cind only if, 
given V at time t', the sequence of effects E{Impy^i.,-j_^p^f_^_-^-j) achieves patt + 1 with probability 1. 

In words, v{t') leads to p{t + 1) with certainty iff the dynamic weight of v{t') equals its static 
weight. This is a simple consequence of how the weight propagation is arranged; it should hold true 
for any reasonable weight propagation scheme ("do not mark a node as certain if it is not"). A full 
proof of the lemma appears in Appendix A on pp. 613. 

Re-consider the second for loop of build-timestep. What happens is the following. Having 
finished the build-w-impleafs weight propagation for p at time t -I- 1, we 

1. collect all the leafs support{p{t + 1)) of Imp^p^^-j that meet the criteria of Lemma 4, and 

2. check (by a call to a SAT solver) whether the knowledge-base formula $ implies the disjunc- 
tion of these leafs. 

If the implication holds, then the examined fact p at time t is added to the set of facts known at time 
t. Finally, the procedure removes from the set of facts that are known to possibly hold at time t + 1 
all those facts that were just proven to hold at time t + 1 with certainty. 

To understand the above, consider the following. With Lemma 4, support{p{t + 1)) contains 
exactly the set of leafs achieving which will lead to p{t + 1) with certainty. Hence we can basically 
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procedure build-w-impleafs {p{t),Imp) 

top-down propagation of weights rop(t) iromp{t) to all nodes in Imp^^^^^-^ 
^p{t) (pit)) ■= 1 

for decreasing time steps t' := {t — 1) . . . (— m) do 
for all chance nodes e{t') G Imp^^^^^^^ do 

a := nreadd(£),r(t'+i)eJmp^^(,) [1 - ^P{t) (rif + 1))] 
^P(t) W)) ■.= w{e{t'))-{l-a) 
endfor 

for all fact nodes q{t') G /mp_^p(() do 
a := 1 

for all a e A(t'),e G £"(0), con(e) = g' do 

1 ~ E£eA(e),£(t')einp^p(t) 



a := a 
endfor 

endfor 
endfor 



Figure 6: The build-w-impleafs procedure for weight back-propagation over the implication graph. 

use the same implication test as in Conformant-FF. Note, however, that the word "basically" in the 
previous sentence hides a subtle but important detail. In difference to the situation in Conformant- 
FF, support{p(t + 1)) may contain two kinds of nodes: (1) proposition nodes at the start layer of 
the PRPG, i.e., at layer —m corresponding to the initial belief; (2) chance nodes at later layers of 
the PRPG, corresponding to outcomes of effects that have no unknown conditions. This is the point 
where the discussed above updates on the formula $ are needed — ^those keep track of alternative 
effect outcomes. Hence testing $ — > \/ iesupport{p{t+i)) ^ same as testing whether either: (1) p 
is known at i + 1 because it is always triggered with certainty by at least one proposition true in the 
initial world; or (2) p is known at t -|- 1 because it is triggered by all outcomes of an effect that will 
appear with certainty. We get the following result: 

Lemma 5 Let {A, Mbj , G, 9) be a probabilistic planning task, a be a sequence of actions applicable 
in bi, and \^ be a relaxation function for A. For each time step t > —m, and each proposition p G 
V, if P{t) is constructed by build-PRPG(a, A, <p{Mbj),G, 9, \ then p at time t can be achieved 
by a relaxed plan starting with o| \ 

(1) with probability > (that is, p is not negatively known at time t) if and only ifp G uP{t)UP{t), 
and 

(2) with probability 1 (that is, p is known at time t) if and only ifp G P{t). 

This is a consequence of the arguments outUned above. The full proof of Lemma 5 is given in 
Appendix A on pp. 614. 

Let us now consider the weight-propagating^ procedure build-w-impleafs depicted in Figure 6. 
This procedure performs a layered, top-down weight propagation from a given node^ p{t) G Imp 

8. The weight propagation scheme of the build-w-impleafs procedure is similar in nature to this used in the heuristics 
module of the recent probabilistic temporal planner Prottle of Little, Aberdeen, and Thiebaux (2005). 

9. Note that the "t" here will be instantiated with t + 1 when called from build-timestep. 
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down to the leafs of Jmp_^p(^) . This order of traversal ensures that each node of Imp^p^^^-^ is pro- 
cessed only after all its successors in Imp^p^^y For the chance nodes s{t'), the dynamic weight 

^p(t) {^{t')) is set to 

1. the probability that the outcome e takes place at time t' given that the corresponding action 
effect e(e) does take place at t', times 

2. an estimate of the probability of achieving p at time t by the effects £'(Imp^(j/)_^p^(^). 

The first quantity is given by the "global", static weight w (e(t')) assigned to e{t') in the first for 
loop of build-timestep. The second quantity is derived from the dynamic weights ^^(f) {r{t' + 1)) 
for r € add(e), computed in the previous iteration of the outermost for loop of build-w-impleafs. 
Making a heuristic assumption that the effect sets E{Iinpj.(^f.,_^_i-^^p(^^-^) for different r G add{e) are 
all pairwise independent, a is then set to the probability of failure to achieve p at t by the effects 
E{Imp^^^,-^^p^f-'j). This computation of a for e{t') is decomposed over the artifacts of e, and this 
is where the weight propagation starts taking place. For the fact nodes q(t'), the dynamic weight 
^p(t) is set to the probability that some action effect conditioned on q at time t' allows 

(possibly indirectly) achieving the desired fact p at time t. Making again the heuristic assumption 
of independence between various such effects conditioned on q at t', computing Wp(f-^ (^(^O) is 
decomposed over the outcomes of these effects. 

procedure get-P (t, G) 

estimates the probabihty of achieving G at time p. 
if G g P{t) U uP{t) then return endif 
if G C P{t) then return 1 endif 

for 5 G G \ P{t) do 

for each / e leafs {Imp ^gf^j.^), introduce a chance proposition {Ig) with weight cc^g(t) (l) 

^9 •= l&leafs{Imp^gi^^)) ^ Al£leafs{Imp^g^^^)r\uP{-m) ("'^ ^ (^fl)) 

endfor 

return WMC($ A AgeG\P(t) Vg) 

Figure 7: Estimating the goal likelihood at a given time step. 

What remains to be explained of the build-PRPG procedure are the two termination criteria of 
the while loop constructing the planning graph from the layer onwards. The first test is made by 
a call to the get-P procedure, and it checks whether the PRPG built to the time layer T contains 
a relaxed plan for {A,J\fbj, G, 6). The get-P procedure is shown in Figure 7. First, if one of the 
subgoals is negatively known at time t, then, from Lemma 5, the overall probability of achieving 
the goal is 0. On the other extreme, if all the subgoals are known at time t, then the probability of 
achieving the goal is 1. The correctness of the latter test is implied by Lemma 5 and non-interference 
of relaxed actions. This leaves us with the main case in which we are uncertain about some of the 
subgoals. This uncertainty is either due to dependence of these subgoals on the actual initial world 
state, or due to achieving these subgoals using probabilistic actions, or due to both. The uncertainty 
about the initial state is fully captured by our weighted CNF formula (j){Afbj) ^ <|). Likewise, the 
outcomes' chance propositions e{t') introduced into the implication graph by the build-timestep 
procedure are "chained up" in Imp to the propositions on the add lists of these outcomes, and 
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"chained down" in Imp to the unknown (relaxed) conditions of these outcomes, if any. Therefore, 
if some action outcome s at time f/ < t is relevant to achieving a subgoal g G G at time t, then 
the corresponding node e{t') must appear in Imp^g^^y and its weight will be back-propagated by 
build-w-impleafs(5(t), Imp) down to the leafs of Imp^gf^^y The get-P procedure then exploits 
these back-propagated estimates by, again, taking a heuristic assumption of independence between 
achieving different subgoals. Namely, the probability of achieving the unknown sub-goals G\P{t) 
is estimated by weighted model counting over the formula conjoined with probabilistic theories 
ipg of achieving each unknown goal g in isolation. To understand the formulas (pg, consider that, in 
order to make g true at t, we must achieve at least one of the leafs I of Imp^g^^^-^ ; hence the left part of 
the conjunction. On the other hand, if we make I true, then this achieves g{t) only with (estimated) 
probability vjg^i^ {I); hence the right part of the conjunction requires us to "pay the price" if we set 
/to true. 1° 

As was explained at the start of this section, the positive PRPG termination test may fire even if 
the real goal probability is not high enough. That is, get-P may return a value higher than the real 
goal probability, due to the approximation (independence assumption) done in the weight propaga- 
tion. Of course, due to the same approximation, it may also happen that get-P returns a value lower 
than the real goal probabiUty. 

The second PRPPG termination test comes to check whether we have reached a point in the 
construction of PRPG that allows us to conclude that there is no relaxed plan for {A, Mbj , G, 9) that 
starts with the given action sequence a. This termination criterion asks whether, from time step t 
to time step t + 1, any potentially relevant changes have occurred. A potentially relevant change 
would be if the goal-satisfaction probability estimate get-P grows, or if the known and unknown 
propositions grow, of if the support leafs of the latter propositions in Imp that correspond to the 
initial belief state grow. ^ ' If none occurs, then the same would hold in all future iterations t' > t, 
implying that the required goal satisfaction probability would never be reached. In other words, 
the PRPG construction is complete. 

Theorem 6 Let {A,Afbj,G, 9) be a probabilistic planning task, a be a sequence of actions appli- 
cable in hi, and be a relaxation function for A. T/' build-PRPG(a, ^, i^(A/6j), G, ^, \i) returns 
FALSE, then there is no relaxed plan for (A, bj,G, 9) that starts with a\]^. 

Note that Theorem 6 holds despite the approximation done during weight propagation, making 
the assumption of probabihstic independence. For Theorem 6 to hold, the only requirement on the 
weight propagation is this: if the real weight still grows, then the estimated weight still grows. This 
requirement is met under the independence assumption. It would not be met under the assumption of 
co-occurence, propagating weights by maximization operations, and thereby conservatively under- 
estimating the weights. With that propagation, if the PRPG fails then we cannot conclude that there 
is no plan for the respective belief. This is another good argument (besides the bad quality heuristics 
we observed empirically) against using the conservative estimation. 

10. If we do not introduce the extra chance propositions {Ig), and instead assign the weight rxjg^t) {I) to I itself, then the 
outcome is not correct: we have to "pay" also for setting I to false. 

11. To understand the latter, note that PRPG can always be added with more and more replicas of probabilistic actions 
irrelevant to achieving the goals, and having effects with known conditions. While these action effects (since they are 
irrelevant) will not influence our estimate of goal-satisfaction probability, the chance propositions corresponding to 
the outcomes of these effects may become the support leafs of some unknown proposition p. In the latter case, the 
set of support leafs support(p{t')) will infinitely grow with t' — t- oc, while the projection of support(p{t')) on the 
initial belief state (that is, support{p{t)) n uP{t)) is guaranteed to reach a fix point. 
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The full proof to Theorem 6 is given in Appendix A on pp. 615. The theorem finalizes our 
presentation and analysis of the process of constructing probabiUstic relaxed planning graphs. 

4.3 Example: PRPG Construction 

To illustrate the construction of a PRPG by the algorithm in Figures 4-7, let us consider a simplifi- 
cation of our running Examples 1-2 in which 

(i) only the actions {move-b-right, move-left} constitute the action set A, 

(ii) the goal is G = {ri, 62}, and the required lower bound on the probabihty of success 9 = 0.9, 

(iii) the initial beUef state 6/ is given by the BN J\fbj as in Example 2, and 

(iv) the belief state ba evaluated by the heuristic function corresponds to the actions sequence 
a = {move-b-right). 

The effects/outcomes of the actions A considered in the construction of PRPG are described in 
Table 3, where e"^'"^ is a re-notation of the effect e in Table 1, the effect e' in Table 1 is effectively 
ignored due to the emptiness of its add effects. 



a 


E{a) 


con{() 


con{c)\^ 


A(r,) 


Pri::) 


acidic) 


^mbr (jjiove-b-right) 


gmbr 


{ri,h} 


{n} 


_mbr 
"1 


0.7 


{'•2,52} 


^mbr 

^2 


0.2 


{^2} 


^mbr 


0.1 





a"^' (move-left) 




{r2} 


{r2} 


^ml 


1.0 


{n} 


noop^j 




{n} 


{ri} 




1.0 


{n} 


nooprj 




{r2} 


{r2} 




1.0 


{^2} 


noopb^ 




{bi} 


{bi} 




1.0 


{^1} 




e'^ 


{b2} 


{b2} 




1.0 





Table 3: Actions and their |^ relaxation for the PRPG construction example. 

The initialization phase of the build-PRPG procedure results in <5 = ^(A/'fej), Imp := 0, 
P(—l) = 0, and uP{—l) = {n, r2, 61, 62}- The content of nP(— 1) is depicted in the first column 
of nodes in Figure 8. The first for loop of build-PRPG (constructing PRPG for the "past" layers 
corresponding to a) makes a single iteration, and calls the build-timestep procedure with t = —1 
and A(-l) = {a""^'} U NOOPS. (In what follows, using the names of the actions we refer to their 
1^ relaxations as given in Table 3.) The add list of the outcome £3^'''' is empty, and thus it adds no 
nodes to the implication graph. Other than that, the chance nodes introduced to Imp by this call to 
build-timestep appear in the second column of Figure 8. The first outer for loop of build-timestep 
results in Imp given by columns 1-3 of Figure 8, uP(0) = uP(—l), and no extension of $. 

In the second outer for loop of build-timestep, the weight propagating procedure build-w-impleafs 
is called for each unknown fact p(0) G uP(0) = {ri(0), r2(o), &i(0), 62(0)}, generating the "p(0)- 
oriented" weights as in Table 4. For each p(0) G uP(0), the set of supporting leafs support(p(0)) = 
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e^^{l)\ ^62(2) 



Figure 8: The implication graph Imp. The odd columns of nodes depict the sets of unknown propo- 
sitions uP{t). The even columns of nodes depict the change propositions introduced for 
the probabilistic outcomes of the actions A{t). 



{p(— 1)}, none of them is imphed by $ = Mi^, and thus the set of known facts -P(O) remains equal 

to P(-l) = 0, and uP(-l) equal to = uP{-l). 







t' -- 


= 










t' 




1 












n 


r2 


hi 


h2 


^mbr 

^1 












ri 


r2 


bi 


h2 


^n(O) 


1 












1 








1 








^r2(0) 




1 






0.7 


0.2 




1 






0.9 


1 






^61(0) 






1 












1 








1 




^62(0) 








1 


0.7 










1 


0.7 






1 



Table 4: The columns in the table correspond to the nodes in the implication graph Imp, and each 
row provides the weights rop(o) for some p(0) G mP(0). An entry in the row of p(0) is 
empty if and only if the node associated with the corresponding column does not belong 
to the implication subgraph Imp_^p^Q-j . 



Having finished with the for loop, the build-PRPG procedure proceeds with the whUe loop that 
builds the "future" layers of PRPG. The test of goal (un)satisficing get-P(0, G) < 6 evaluates to 
TRUE as we get get-P(0,G) = 0.63 < 0.9, and thus the loop proceeds with its first iteration. 
To see the former, consider the implication graph Imp constructed so far (columns 1-3 in Fig- 
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lire 8). For our goal G = {n, 62} we have leafs{Imp^j._^^Q^) = {ri(— 1)}, and leafs{Imp^^^j^Q^) 
{ri(-l), 62(-l)}. As {ri(0), 62(0)} C uP(0) and $ = (/>(A4,), we have 



where 



and 



get-P(0, G) = WMC A ipr, A , 

V'ri = ((ri,ri» A (ri O {ri^n)) , 

Vb, = V (62,62)) A (ri(-l) ^ (ri,6,)) A (62(-l) ^ (62,62)) , 

^((n.ri)) = -zJ^nCo) (n(-l)) = 1 

((62,62)) = t^(,2(0)(62(-l)) = l • (21) 
^((^,62)) = ^62(0) (n(-l)) = 0.7 

Observe that the two models of (t){J\fbj) consistent with r2 immediately falsify the sub-formula 
(t){J^bi) A ipn- Hence, we have 

get-P(0,G) = \NMC{(t)iMb,)AVr^/\'Pb2\r^(-i)=iMi-i)=i) + 

WMC {^{Mbj) A (pri A ipb2\r,(-l)=l,b2{-l)=l) 

= bi{ri,bi) •ro((ri,ri)) ■ {{ri,b2)) + bi{ri,b2) ■ w {{n^n)) •^((''1,62)) • ((^2,62)) 
= 0.63 •1-0.7 + 0.27-1 -0.7 -1 
= 0.63 

In the first iteration of the whUe loop, build-PRPG calls the build-timestep procedure with 

t = and ^(0) = {a""^' , a""'} U NOOPS. The chance nodes introduced to Imp by this call to 
build-timestep appear in the forth column of Figure 8. The first outer for loop of build-timestep 
results in Imp given by columns 1-5 of Figure 8, uP{l) = uP(0), and no extension of As 
before, in the second for loop of build-timestep, the build-w-impleafs procedure is called for each 
unknown fact G uP{l) = {ri(l), r2(i) , 6i(l), 62(1)}, generating the "p(l)-oriented" weights. 
The interesting case here is the case of weight propagation build-w-impleafs(ri(l), Imp), resulting 
in weights 



•i(i)(n(i)) = 1 



^'^^ ~ -n(i)(^r^(-l)) = 0.7 ^.,(i)(r2(-l)) = 1 

^n(i)(^2'^(-l)) = 0.2 



zu, 



(i)(e''H-i)) = 1 



^ri(i)(n(o)) = 1 

^ri{l){r2{0)) = 1 



for the nodes in Jmp_j.^^^) . From that, the set of supporting leafs of ri (1) is assigned to support{ri{l)) 
{ri(— l),r2(— 1)}, and since $ = (f){Afbi) does impUes ri(— 1) V r2(— 1), the fact ri is concluded 
to be known at time 1, and is added to P(l). For all other nodes G uP{l) we still have 
support{p{l)) = {p(— 1)}, and thus they all remain unknown at time t = 1 as well. Putting 
things together, this call to the build-w-impleafs procedure results with P{1) = {ri(l)}, and 
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uP{l) = {''2(1)) 62(1)}- The while loop of the build-PRPG procedure proceeds with check- 
ing the fixpoint termination test, and this immediately fails due to -P(l) ^ -P(O)- Hence, the while 
loop proceeds with the next iteration corresponding to t = 1. 

The test of goal (un)satisficing get-P(l,G) < 9 still evaluates to TRUE because we have 
get-P(l, G) = 0.899 < 0.9. Let us follow this evaluation of get-P(l, G) in detail as well. Consid- 
ering the implication graph Imp constructed so far up to time t = 1 (columns 1-5 in Figure 8), and 
having G n uP{l) = {62(1)}, leafs {Imp^^^^^^) = {ri(-l), 62(-l)}, and (still) $ = we 
obtain 

get-P(l,G)=WMC(0(A4,)Ay.6j, 

with 

^b, = {{tim) V (^2,62)) A (ri(-l) o (ri,(.,)) A (62(-l) ^ (62,62)) > (22) 

While the structure of ipb2 in Equation 22 is identical to this in Equation 20, the weights associated 
with the auxiliary chance propositions are different, notably 

^ ((62,62)) = -^62(1) (^2(-l)) = 1 

^((n,62)) = ^Mi) (ri(-l)) = 0.9l 

The difference in zu {{ri^b2)) between Equation 21 and Equation 23 stems from the fact that ri(— 1) 
supports 62(1) not only via the effect e"^'^'' at time —1 but also via the a different instance of the 
same effect at time 0. Now, the only model of (p{Nbi) that falsify </?b2 is the one that sets both ri 
and 62 to false. Hence, we have 

get-P(l,G) = 6/(ri,6i)-ro((ri,52)) + 

hi{riM) • w((n,fe2)) ■^((62,62)) + 
67(^2,62) •ro((62,fe2)) 

= 0.63 -0.91 +0.27- 0.91- 1 + 0.08-1 

= 0.899 

Having verified get-P(l, G) < 6, the while loop proceeds with the construction for time t = 2, 
and calls the build-timestep procedure with t = 1 and A{1) = {a""^', a"'^}UNOOPS. The chance 
nodes introduced to Imp by this call to build-timestep appear in the sixth column of Figure 8. The 
first outer for loop of build-timestep results in Imp given by columns 1-7 of Figure 8, and 

A (-e'T'^'Cl) V -£2 '"■(1)) A (-£!r'"'(l) V -£3 '"■(1)) A (-£3 '"'(0) V -£3 """(O)) 

(24) 

Next, the build-w-impleafs procedure is called as usual for each unknown fact p(2) G uP{2) = 
{r2(2), 61(2), 62(2)}. The information worth detailing here is that now we have leafs{Imp_^^^^2)) = 
{ 62 ( - 1 ) , ri ( - 1 ) , e';"'"' ( 1 ) } , and support{b2 (2)) = {62(-l),e',"'"'(l)}. However, we still have $ 
y iesupport{p{2)) ^ f^'" '^^ pC^) ^ uP{2), and thus the set of known facts P(2) remains equal to 

P(i) = in}. 
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Returning from the call to the build-w-impleafs procedure, build-PRPG proceeds with checking 
the fixpoint termination condition. This time, the first three equalities of the condition do hold, yet 
the condition is not satisfied due to get-P(2, G) > get-P(i, G). To see the latter, notice that we have 

get-P(2,G) =WMC($A^6j, 

where $ is given by Equation 24, 

'fh, = {{ri,b,) V (62,6,) V £^^''^(1)) A (ri(-l) ^ (n,,,)) A (62(-l) ^ (?>2,te)) , (25) 

and 

^((^2,62)) 
^(^62)) 

It is not hard to verify that 

get-P(2,G) = get-P(l,G) + 6/(r2,6i)-tu(eJ^'^^(l)) 
= 0.899 + 0.02-0.7 
= 0.913 

Note that now we do have get-P(2, G) > 9, and therefore build-PRPG aborts the while loop by 
passing the goal satisficing test, and sets T = 2. This finalizes the construction of PRPG, and thus, 
our example. 

4.4 Extracting a Probabilistic Relaxed Plan 

If the construction of the PRPG succeeds in reaching the goals with the estimated probability of suc- 
cess get-P(T, G) exceeding 6, then we extract a relaxed plan consisting of A' C ^4(0), . . . , A{T — 
1), and use the size of A' as the heuristic value of the evaluated belief state ha- 

Before we get into the technical details, consider that there are some key differences between 
relaxed (no delete lists) probabilistic planning on the one hand, and both relaxed classical and re- 
laxed qualitative conformant planning on the other hand. In relaxed probabilistic planning, it might 
make sense to execute the same action numerous times in consecutive time steps. In fact, this 
might be essential - just think of throwing a dice in a game until a "6" appears. In contrast, in the 
relaxed classical and qualitatively uncertain settings this is not needed - once an effect has been 
executed, it remains true forever. Another complication in probabilistic planning is that the required 
goal-achievement probability is specified over a conjunction (or, possibly, some more complicated 
logical combination) of different facts. While increasing the probability of achieving each individ- 
ual sub-goal (/ G G in relaxed planning will always increase the overall probability of achieving G, 
choosing the right distribution of effort among the sub-goals to pass the required threshold 9 for the 
whole goal G is a non-trivial problem. 

A fundamental problem is the aforementioned lack of guarantees of the weight propagation. 
On the one hand, the construction of PRPG and Lemma 5 imply that a|^ concatenated with an 
arbitrary linearization of ^(0), . . . , A(T — 1) is executable in hi. On the other hand, due to 
the independence assumption made in the build-w-impleafs procedure, get-P(T, G) > 9 does not 



= ^Mi)(^2(-l)) = l 

= ^&2(i) (n(-l)) = 0.91 (26) 
= ^Mi)(^5"''(l))=0-7 



593 



DoMSHLAK & Hoffmann 



imply that the probabihty of achieving G by a\'{ concatenated with a exceeds 9. A "real" relaxed 
plan, in that sense, might not even exist in the constructed PRPG. 

Our answer to the above difficulties is to extract relaxed plans that are correct relative to the 
weight propagation. Namely, we use an implication graph "reduction" algorithm that computes 
a minimal subset of that graph which still - according to the weight propagation - sufficiently 
supports the goal. The relaxed plan then corresponds to that subset. Obviously, this "solves" the 
difficulty with the lack of "real" relaxed plans; we just do the relaxed plan extraction according to 
the independence assumption (besides ignoring deletes and removing all but one condition of each 
effect). The mechanism also naturally takes care of the need to apply the same action several times: 
this corresponds to several implication graph edges which are all needed in order to obtain sufficient 
weight. The choice of how effort is distributed among sub-goals is circumvented in the sense that 
all sub-goals are considered in conjunction, that is, the reduction is performed once and for all. Of 
course, there remains a choice in which parts of the impUcation graph should be removed. We have 
found that it is a useful heuristic to make this choice based on which actions have already been 
applied on the path to the behef . We will detail this below. 

Making another assumption on top of the previous relaxations can of course be bad for heuristic 
quality. The "relaxed plans" we extract are not guaranteed to actually achieve the desired goal prob- 
ability. Since the relaxed plans are used only for search guidance, per se this theoretical weakness is 
only of marginal importance. However, an over-estimation of goal probability might result in a bad 
heuristic because the relaxed plan does not include the right actions, or does not apply them often 
enough. In Section 5, we will discuss an example domain where Probabihstic-FF fails to scale for 
precisely this reason. 

Figure 9 shows the main routine extract-PRPIan for extracting a relaxed plan from a given 
PRPG (note that T is the index of the highest PRPG layer, c.f. Figure 4). The sub-routines of 
extract-PRPIan are shown in Figures 10-1 1. At a high level, the extract-PRPIan procedure consists 
of two parts: 

1 . Reduction of the impUcation graph, aiming at identifying a set of time-stamped action effects 
that can be ignored without decreasing our estimate of goal-achievement probabihty get-P(r, G) 
below the desired threshold 0, and 

2. Extraction of a valid relaxed plan If such that (schematically) constructing PRPG with instead 
of the full set of ^(0), . . . , A{T) would still resuh in get-P(T, G) > 9. 

The first part is accomplished by the reduce-implication-graph procedure, depicted in Figure 10. 
As of the first step in the algorithm, the procedure considers only the parts of the implication graph 
that are relevant to achieving the unknown sub-goals. Next, reduce-implication-graph performs a 
greedy iterative ehmination of actions from the "future" layers 0, . . . , T— 1 of PRPG until the proba- 
bihty estimate get-P(T, G) over the reduced set of actions goes below 9. While, in principle, any ac- 
tion from ^(0), . . . , A{T—1) can be considered for elimination, in reduce-implication-graph we ex- 
amine only repetitions of the actions that already appear in a. Specifically, reduce-implication-graph 
iterates over the actions a in alf, and if a repeats somewhere in the "future" layers of PRPG, then 
one such repetition a{t') is considered for removal. If removing this repetition of a is found safe 
with respect to achieving 9,^^^ then it is effectively removed by eliminating all the edges in Imp that 
are induced by a{t'). Then the procedure considers the next repetition of a. If removing another 

12. Note here that the formula for WMC is constructed exactly as for the get-P function, c.f. Figure 7. 
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procedure extract-PRPIan fPi^PGfa, A, 4>(Nb, ),G, 9, \t)), 

selects actions from A{0), . . . , A{T — 1) 
Imp' ■= reduce-implication-graph() 
extract-subplan(7mp') 
sub-goal(GnP(T)) 

for decreasing time steps f := T, . . . , 1 do 
for all g G G{t) do 

if 3a e A{t- l),e € E{a),con{e) S P{t - 1),V£ e A(e) : g € add{s) then 

add-to-relaxed-plan one such a at time t 

sub-goal(pre(a) U con(e)) 
else 

Imp^^*^ := construct-support-graph(support(5(t))) 
extract-su bplan (Imp ^^*^) 
endif 
endfor 
endfor 



Figure 9: Extracting a probabilistic relaxed plan. 

copy of a is not safe anymore, then the procedure breaks the inner loop and considers the next 
action. 

procedure reduce-implication-graph() 
operates on the PRPG; 
returns a sub-graph of Imp. 

Imp' := \Jg(,G\P(T)linp^g(T) 
for all actions o G do 

for all edges {s{t'),p{t' + 1)) G Imp", induced by a{t') G A{t'), for some t' >0do 
Imp" := Imp' 

remove from Imp" all the edges induced by a G A(t') 
for all 5 G G \ P{t) do 

for each I G leafs{Imp" ^g^j^), introduce a chance proposition {Ig) with weight ^^g(T) (0 

iPg ■= iy leleafs{Imi/' ^ l\leleafs{Imp" ^g^j.))nuP{-m) ("'^ ^ {Ig)) 

endfor 

if WMC($ A l\geG\p{T) Vg) > ^ then Imp' := Imp" else break endif 
endfor 
endfor 
return Imp' 

Figure 10: The procedure reducing the impUcation graph. 

To illustrate the intuition behind our focus on the repetitions of the actions from a, let us con- 
sider the following example of a simple logistics-style planning problem with probabilistic actions. 
Suppose we have two locations A and B, a truck that is known to be initially in A, and a heavy and 
uneasy to grab package that is known to be initially on the truck. The goal is to have the package 
unloaded in B with a reasonably high probability, and there are two actions we can use - moving 
the truck from Ato B (a"^), and unloading the package (a"). Moving the truck does not necessarily 
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move the truck to B, but it does that with an extremely high probability. On the other hand, unload- 
ing the bothersome package succeeds with an extremely low probability, leaving the package on the 
truck otherwise. Given this data, consider the behef state ba corresponding to "after trying to move 
the truck once", that is, to the action sequence (a"^). To achieve the desired probability of success, 
the PRPG will have to be expanded to a very large time horizon T, allowing the action a" to be 
applied sufficiently many times. However, the fact "truck in B" is not known in the belief state ba, 
and thus the implication graph will also contain the same amount of apphcations of a"^. Trimming 
away most of these applications of a"^ will still keep the probability sufficiently high. 

The reader might ask at this point what we hope to achieve by "trimming away most of the 
applications of a"^". The point is, intuitively, that the implication graph reduction mechanism is 
a means to understand what has been accomplished already, on the path to ha- Without such an 
understanding, the relaxed planning can be quite indiscriminative between search states. Consider 
the above example, and assume we have not one but two troubled packages, PI and P2, on the 
truck, with unload actions a"^ and a"^. The PRPG for ba contains copies of a"^ and a"^ at layers up 
to the large horizon T. Now, say our search starts to unload PI. In the resulting belief, the PRPG 
still has T steps because the situation has not changed for P2. Each step of the PRPG still contains 
copies of both a"^ and a"^ - and hence the heuristic value remains the same as before! In other 
words, without an implication graph reduction technique, relevant things that are accomplished 
may remain hidden behind other things that have not yet been accomplished. In the above example, 
this is not really critical because, as soon as we have tried an unload for each of PI and P2, the 
time horizon T decreases by one step, and the heuristic value is reduced. It is, however, often 
the case that some sub-task must be accomplished before some other sub-task can be attacked. In 
such situations, without impUcation graph reduction, the search staggers across a huge plateau until 
the first task is completed. We observed this in a variety of benchmarks, and hence designed the 
implication graph reduction to make the relaxed planning aware of what has already been done. 

Of course, since our weight propagation may over-estimate true probabilities, and hence over- 
estimate what was achieved in the past, the implication graph reduction may conclude prematurely 
that a sub-task has been "completed". This leads us to the main open question in this research; we 
will get back to this at the end of Section 5, where we discuss this in the context of an example 
where Probabilistic-FF's performance is bad. 

Let us get back to explaining the extract-PRPIan procedure. After the imphcation graph reduc- 
tion, the procedure proceeds with the relaxed plan extraction. The process makes use of proposition 
sets . . . , G{T), which are used to store time-stamped sub-goals arising at layers I <t <T 
during the relaxed plan extraction. The sub-routine extract-subplan (Figure 11) 

1. adds to the constructed relaxed plan all the time-stamped actions responsible for the edges of the 
reduced implication graph Imp', and 

2. subgoals everything outside the implication graph that condition the applicability of the effects 
responsible for the edges of Imp'. 

Here and in the later phases of the process, the sub-goals are added into the sets G(l), . . . , G{T) by 
the sub-goal procedure that simply inserts each given proposition as a sub-goal at the first layer of 
its appearance in the PRPG. Having accompUshed this extract-and-subgoal pass of extract-subplan 
over Imp' , we also subgoal all the goal conjuncts known at time T. 

In the next phase of the process, the sub-goals are considered layer by layer in decreasing order 
of time steps T >t>l. For each sub-goal g at time t, certain supporting actions are selected into 
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procedure extract-subplan(/jnp') 

actions that are helpful for achieving uncertain goals G fl uP{T) and 
subgoals all the essential conditions of these actions 
for each edge {e{t),p{t + 1)) G Imp' such that t > do 

if action a and its effect e G E{a) be responsible for e at time t time 
add-to-relaxed-plan a at time t 
sub-goal ((pre (a) U con(e)) fl P{t)) 
endif endfor 

procedure sub-goal (P) 

inserts the propositions in P as sub-goals 
at the layers of their first appearance in the PRPG 
for all p e P do 

to := aigmin^ {p £ P{t)} 
if to > 1 then G{to) := G{to) U {p} endif 
endfor 

procedure construct-support-graph(support(5(t))) 

takes a subset support{g{t)) of leafs{Imp_^g^^^) weighted according to g{t); 
returns a sub-graph Imp' of Imp. 

Imp' := 

open := support{g{t)) 
wliile open 7^ do 

open := open \ {p{t')} 

choose a £ A{t'), e e E{a), con{e) = {p} such that 

V£ e A(e) : (p(t'), £(*')) e /mp^^,) A tUg(t)(£(t')) = ^^(^(i')) 
for each e e A(e) do 

choose q e add{e) such that Wg(t){q{t' + 1)) = 1 
Imp' := imp' U {{p{t'), e{t')), {e{t'), q{t' + 1))} 
open := open U {q{t' + 1)} 
endfor endwiiile 
return Imp' 

Figure 11: Sub-routines for extract-PRPIan. 

the relaxed plan. If there is an action a and some effect e G £'(a) that are known to be applicable 
at time t — 1, and guarantee to achieve g with certainty, then a is added to the constructed relaxed 
plan att — 1. Otherwise, we 

1. use the construct-support-graph procedure to extract a sub-graph Imp^^*^ consisting of a set of 
implications that together ensure achieving g at time t, and 

2. use the already discussed procedure extract-subplan to 

(a) add to the constructed relaxed plan all the time-stamped actions responsible for the edges 
of Imp ^W, and 

(b) subgoal everything outside this implication graph Imp that condition the applicability 
of the effects responsible for the edges of Imp . 
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Processing this way all tlie sub-goals down to G{1) finalizes the extraction of the relaxed plan 
estimate. Section 4.5 provides a detailed illustration of this process on the PRPG constructed in 
Section 4.3. In any event, it is easy to verify that the relaxed plan we extract is sound relative to our 
weight propagation, in the following sense. 

Proposition 7 Let {A,Mbj, G, 9) be a probabilistic planning task, a be a sequence of actions ap- 
plicable in bj, and \ f be a relaxation function for A such that build-PRPG(a, A, 0(A4^), G, 9, |^) 
returns TRUE. Let A(0)*, . . . , A{T - ly be the actions selected from A{0), ... , A{T - 1) by 
extract-PRPIan. When constructing a relaxed planning graph using only AiOy, . . . , A{T — ly, 
then get-P {T,G) > 9. 

Proof: By construction: reduce-implication-graph leaves enough edges in the graph so that the 
weight propagation underlying get-P still concludes that the goal probability is high enough. ■ 

4.5 Example: Extracting a Relaxed Plan from PRPG 

We illustrate the process of the relaxed plan extraction on the PRPG as in Figure 8, constructed for 
the belief state and problem specification as in example in Section 4.3. In this example we have 
T = 2, G n uP{2) = {62 }» and thus the implication graph Imp gets immediately reduced to its 
sub-graph Imp' depicted in Figure 12a. As the plan a to the belief state in question consists of only a 
single action a'^''^ the only action instances that are considered for elimination by the outer for loop 
of reduce-implication-graph are a"^'"'(0) and a'^'"'(l). If a'^'"'(0) is chosen to be examined, then the 
implication sub-graph Imp" = Imp' is further reduced by removing all the edges due to a'^'"^(0), 
and the resulting Imp" appears^-' in Figure 12b. The ^ and 99^2 components of the evaluated formula 
$ A </?b2 are given by Equation 24 and Equation 25, respectively, and the weights associated with 
the chance propositions in Equation 25 over the reduced implication graph Imp" are 

^62(1) (&2(-l)) = 1 

^62(1) (n(-i)) = 0.7 . (27) 
t^6.(i)(£r(l)) = 0.7 

The weight model counting of $ A ^ph2 evaluates to 0.724 < 9, and thus Imp" does not replace Imp' . 
The only alternative action removal is this of a"^^'{l), and it can be seen from the example in Sec- 
tion 4.3 that this attempt for action elimination will also result in probability estimate lower than 9. 
Hence, the only effect of reduce-implication-graph on the PRPG processed by the extract-PRPIan 
procedure is the reduction of the implication graph to only the edges relevant to achieving {62} at 
time T = 2. The reduced implication sub-graph Imp' returned by the reduce-implication-graph 
procedure is depicted in Figure 12a. 

Next, the extract-subplan procedure iterates over the edges of Imp' and adds to the initially 
empty relaxed plan applications of a"^'"' at times and 1. The action a"^^'' has no preconditions, 
and the condition ri of the effect e™^^ G E{a"^^'') is known at time 1. Hence, extract-subplan 
invokes the sub-goal procedure on {r (1)}, and the latter is added into the proposition set G(l). The 
subsequent call sub-goal(G fl P{T)) = sub-goal({ri}) leads to no further extensions of G{2), G{1) 

13. The dashed edges in Figure 12b can be removed from Imp" either now or at a latter stage if Imp" is chosen to replace 
Imp'. 



((62,62)) = 
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Figure 12: Illustrations for various steps of the relaxed plan extraction from the PRPG constructed 
in Section 4.3, and, in particular, from the implication graph of the latter, depicted in 
Figure 8. 



as we already have ri G G{1). Hence, the outer for loop of extract-PRPIan starts with G{2) = 0, 
andG(l) = {ri}. 

Since G{2) is empty, the first sub-goal considered by extract-PRPIanis ri from G(l). For ri 
at time 1, no action effect at time passes the test of the if statement — ^the condition r2 of £'"' 
is not known at time 0, and the same is true^^ for e^^. Hence, the subgoal ri(l) is processed 
by extracting a sub-plan to support achieving it with certainty. First, the construct-support-graph 
procedure is called with support{r = {ri(— 1), r2(— 1)} (see Section 4.3). The extracted sub- 

14. In fact, it is easy to see from the construction of the sub-goal procedure that if p belongs to G{t), then the condition 
of the noop's effect s** cannot be known at time t — 1. 
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graph Imp'^^^^^ of the original implication graph Imp is depicted in Figure 12c, and invoking the 
procedure extract-subplan on Imp''"^^^^ results in adding (i) application of a"^' at time 0, and (ii) 
no new subgoals. Hence, the proposition sets G(1),G(2) get emptied, and thus we end up with 
extracting a relaxed plan {a"'^'{0),a"'\0),a"'^'{l)). 

5. Empirical Evaluation 

We have implemented Probabilistic-FF in C, starting from the Conformant-FF code. With 6 = 1.0, 
Probabilistic-FF behaves exactly like Conformant-FF (except that Conformant-FF cannot handle 
non-deterministic effects). Otherwise, Probabilistic-FF behaves as described in the previous sec- 
tions, and uses Cachet (Sang et al., 2005) for the weighted model counting. To better home in on 
strengths and weaknesses of our approach, the empirical evaluation of ProbabiUstic-FF has been 
done in two steps. In Section 5.1 we evaluate Probabilistic-FF on problems having non- trivial un- 
certain initial states, but only deterministic actions. In Section 5.2 we examine Probabilistic-FF 
on problems with probabilistic action effects, and with both sources of uncertainty. We compare 
Probabilistic-FF' s performance to that of the probabilistic planner POND (Bryce et al., 2006). The 
reasons for choosing POND as the reference point are twofold. First, similarly to Probabilistic-FF, 
POND constitutes a forward-search planner guided by a non-admissible heuristic function based 
on (relaxed) planning graph computations. Second, to our knowledge, POND clearly is the most 
efficient probabiUstic planner reported in the literature. 

The experiments were run on a PC running at 3GHz with 2GB main memory and 2MB cache 
running Linux. Unless stated otherwise, each domain/problem pair was tried at four levels of de- 
sired probability of success 6 G {0.25, 0.5, 0.75, 1.0}. Each run of a planner was time-limited by 
1800 seconds of user time. ProbabiUstic-FF was run in the default configuration inherited from FF, 
performing one trial of enforced hill-climbing and switching to best-first search in case of failure. In 
domains without probabilistic effects, we found that Probabilistic-FF's simpler relaxed plan extrac- 
tion developed for that case (Domshlak & Hoffmann, 2006), performs better than the one described 
in here. We hence switch to the simpler version in these domains. 

Unlike Probabilistic-FF, the heuristic computation in POND has an element of randomization; 
namely, the probability of goal achievement is estimated via sending a set of random particles 
through the relaxed planning graph (the number of particles is an input parameter). For each prob- 
lem instance, we averaged the runtime performance of POND over 10 independent runs. In special 
cases where POND timed out on some runs for a certain problem instance, yet not on all of the 
10 runs, the average we report for POND uses the lower-bounding time threshold of 1800s to re- 
place the missing time points. In some cases, POND's best-case performance differs a lot from 
its average performance; in these cases, the best-case performance is also reported. We note that, 
following the suggestion of Dan Bryce, POND was run in its default parameter setting, and, in par- 

15. In our experiments we have used a recent version 2.1 of POND that significantly enhances POND2.0 (Bryce et al., 
2006). The authors would like to thank Dan Bryce and Rao Kambhampati for providing us with a binary distribution 
ofPOND2.1. 

16. Without probabilistic effects, relaxed plan extraction proceeds very much like in Conformant-FF, with an additional 
straightforward backchaining selecting support for the unknown goals. The more complicated techniques developed 
in here to deal with relaxed plan extraction under probabilistic effects appear to have a more unstable behavior than 
the simpler techniques. If there are probabilistic effects, then the simple backchaining is not meaningful because it 
has no information on how many times an action must be applied in order to sufficiently support the goal. 
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Table 5: Empirical results for problems with probabilistic initial states. Times t in seconds, search 
space size |5| (number of calls to the heuristic function), plan length /. 



ticular, this includes the number of random particles (64) selected for computing POND's heuristic 
estimate (Bryce et al., 2006). 

5,1 Initial State Uncertainty and Deterministic Actions 

We now examine the performance of Probabilistic-FF and POND in a collection of domains with 
probabilistic initial states, but with deterministic action effects. We will consider the domains one 
by one, discussing for each a set of runtime plots. For some of the problem instances. Table 5 shows 
more details, providing features of the instance size as well as detailed results for ProbabiUstic-FF, 
including the number of explored search states and the plan length. 

Our first three domains are probabilistic versions of traditional conformant benchmarks: "Safe", 
"Cube", and "Bomb". In Safe, out of n combinations one opens the safe. We are given a probability 
distribution over which combination is the right one. The only type of action in Safe is trying a 
combination, and the objective is to open the safe with probability > 6. We experimented with 
two probability distributions over the n combinations, a uniform one ("Safe-uni") and a distribution 
that declines according to a cubic function ("Safe-cub"). Table 5 shows that Probabilistic-FF can 
solve this very efficiently even with n = 70. Figure 13 compares between Probabilistic-FF and 
POND, plotting their time performance on an identical linear scale, where x-axes show the number 
of combinations. 

From the graphs it is easy to see that ProbabiUstic-FF outperforms POND by at least an order of 
magnitude on both Safe-uni and Safe-cub. But a more interesting observation here is not necessarily 
the difference in time performance, but the relative performance of each planner on Safe-uni and 
Safe-cub. Note that Safe-cub is somewhat "easier" than Safe-uni in the sense that, in Safe-cub, fewer 
combinations must be tried to guarantee a given probabiUty 9 of opening the safe. This because the 
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(a) Uniform prior distribution over the combinations. 
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(b) Cubic decay prior distribution over the combinations. 



Figure 13: The Safe domain, ProbabiUstic-FF (left) vs. POND (right). 



dominant part of the probability mass lies on the combinations at the head of the cubic distribution 
(the last combination has probability to be the right combination, and thus it needs not be tried 
even when 9 = 1.0). The question is now whether the heuristic functions of Probabilistic-FF and 
POND exploit this difference between Safe-uni and Safe-cub. Table 5 and Figure 13 provide an 
affirmative answer for this question for the heuristic function of Probabilistic-FF. The picture with 
POND was less clear as the times spent by POND on (otherwise identical) instances of Safe-uni and 
Safe-cub were roughly the same.'^ 

Another interesting observation is that, for both Probabilistic-FF and POND, moving from 6 = 
1.0 to < 1.0, that is, from planning with qualitative uncertainty to truly probabilistic planning, 

17. On Safe-cub with n = 70 and 9 £ {0.75, 1.0}, POND undergoes an exponential blow-up that is not shown in the 
graphs since these data points would obscure the other data points; anyway, we believe that this blow-up is due only 
to some unfortunate troubles with numerics. 
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(a) Uniform prior distribution over the initial position. 
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(b) Cubic decay prior distribution over the initial position. 



Figure 14: The Cube domain, Probabilistic-FF (left) vs. POND (right). 



typically did not result in a performance decline. We even get improved performance (except for 
6 = 0.75 in Safe-uni). The reason seems to be that the plans become shorter. This trend can be 
observed also in most other domains. The trend is particularly remarkable for Probabilistic-FF, since 
moving from 9 = 1.0 to ^ < 1.0 means to move from a case where no model counting is needed 
to a case where it is needed. (In other words, Probabilistic-FF automatically "specializes" itself for 
the qualitative uncertainty, by not using the model counting. To our knowledge, the same is not true 
of POND, which uses the same techniques in both cases.) 

In Cube, the task is to move into a comer of a 3-dimensional grid, and the actions correspond 
to moving from the current cube cell to one of the (up to 6) adjacent cube cells. Again, we created 
problem instances with uniform and cubic distributions (over the initial position in each dimension), 
and again, Probabilistic-FF scales well, easily solving instances on a 15 x 15 x 15 cube. Within 
our time hmit, POND was capable of solving Cube problems with cube width < 13. Figure 14 
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compares between Probabilistic-FF and POND in more detail, plotting their time performance on 
different linear scales (with x-axes capturing the width of the grid in each dimension), and showing 
at least an order of magnitude advantage for Probabilistic-FF. Note that, 

• Probabilistic-FF generally becomes faster with decreasing 6 (with decreasing hardness of 
achieving the objective), while 6 does not seem to have a substantial effect on the performance 
of POND, 

• Probabilistic-FF exploits the relative easiness of Cube-cub (e.g., see Table 5), while the time 
performance of POND on Cube-cub and Cube-uni is qualitatively identical. 

We also tried a version of Cube where the task is to move into the grid center. Probabilistic-FF is 
bad at doing so, reaching its performance limit din = 7. This weakness in the Cube-center domain 
is inherited from Conformant-FF. As detailed by Hoffmann and Brafman (2006), the reason for the 
weakness hes in the inaccuracy of the heuristic function in this domain. There are two sources of 
this inaccuracy. First, to solve Cube-center in reality, one must start with moving into a comer in 
order to establish her position; in the relaxation, without delete lists, this is not necessary. Second, 
the relaxed planning graph computation over-approximates not only what can be achieved in future 
steps, but also what has already been achieved on the path to the considered belief state. For even 
moderately long paths of actions, the relaxed planning graph comes to the (wrong) conclusion that 
the goal has already been achieved, so the relaxed plan becomes empty and there is no heuristic 
information. 

Next we consider the famous Bomb-in- the-Toilet domain (or Bomb, for short). Our version 
of Bomb contains n bombs and m toilets, where each bomb may be armed or not armed indepen- 
dently with probability 1/n, resulting in huge numbers of initially possible world states. Dunking a 
bomb into an unclogged toilet disarms the bomb, but clogs the toilet. A toilet can be unclogged by 
flushing it. Table 5 shows that Probabilistic-FF scales nicely to n = 50, and becomes faster as m 
increases. The latter is logical and desirable as having more toilets means having more "disarming 
devices", resulting in shorter plans needed. Figures 15 and 16 compare between Probabilistic-FF 
and POND, plotting the time performance of Probabilistic-FF on a linear scale, and that of POND 
on a logarithmic scale. The four pairs of graphs correspond to four choices of number of toilets 
m € {50, 10, 5, 1}. The x-axes in all these graphs correspond to the number of potentially armed 
bombs, where we checked problems with n G {5, 10, 25, 50}. Figure 15 shows that this time 
Probabilistic-FF is at least four orders of magnitude faster than POND; At the extremes, while the 
hardest combination of n = 50, m = 1, and 9 = 0.75 took Probabihstic-FF less than 7 seconds, 
POND timed-out on most of the problem instances. In addition, 

• In Bomb as well, Probabilistic-FF exhibit the nice pattern of improved performance as we 
move from non-probabilistic {9 = 1.0) to probabilistic planning (specifically, 9 < 0.5; for 
< 0.25, the initial state is good enough already). 

• While the performance of Probabilistic-FF improves with the number of toilets, POND seems 
to exhibit the inverse dependence, that is, being more sensitive to the number of states in the 
problem (see Table 5) rather to the optimal solution depth. 

Finally, we remark that, though length-optimality is not explicitly required in probabilistic confor- 
mant planning, for all of Safe, Cube, and Bomb, Probabihstic-FF's plans are optimal (the shortest 
possible). 
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Figure 15: The Bomb domain, ProbabiUstic-FF (left) vs. POND (right). 



Our next three domains are adaptations of benchmarks from deterministic planning: "Logistics", 
"Grid", and "Rovers". We assume that the reader is familiar with these domains. Each Logistics-x 
instance contains 10 cities, 10 airplanes, and 10 packages, where each city has x locations. The 
packages are with chance 0.88 at the airport of their origin city, and uniformly at any of the other 
locations in that city. The effects of all loading and unloading actions are conditional on the (right) 
position of the package. Note that higher values of x increase not only the space of world states, but 
also the initial uncertainty. Grid is the complex grid world run in the AIPS'98 planning competi- 
tion (McDermott, 1998), featuring locked positions that must be opened with matching keys. Each 
Grid-x here is a modification of instance nr. 2 (of 5) run at AIPS'98, with a 6 x 6 grid, 8 locked 
positions, and 10 keys of which 3 must be transported to a goal position. Each lock has x possi- 
ble, uniformly distributed shapes, and each of the 3 goal keys has x possible, uniformly distributed 
initial positions. The effects of pickup-key, putdown-key, and open-lock actions are conditional. 
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(d) 1 toilet 



Figure 16: The Bomb domain, ProbabiUstic-FF (left) vs. POND (right). 



Finally, our last set of problems comes from three cascading modifications of instance nr. 7 (of 
20) of the Rovers domain used at the AIPS'02 planning competition. This problem instance has 6 
waypoints, 3 rovers, 2 objectives, and 6 rock/soil samples. From Rovers to RoversPPP we modify 
the instance/domain as follows. 

• Rovers is the original AIPS'02 problem instance nr. 7, and we use it hear mainly for compar- 
ison. 

• In Rovers?, each sample is with chance 0.8 at its original waypoint, and with chance 0.1 
at each of the others two waypoints. Each objective may be visible from 3 waypoints with 
uniform distribution (this is a probabilistic adaptation of the domain suggested by Bryce & 
Kambhampati, 2004). 
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Figure 17: Probabilistic-FF and POND on problems from (a) Sand-Castle, and (b) SUppery- 
Gripper. 



• RoversPP enhances RoversP by conditional probabilities in the initial state, stating that whether 
or not an objective is visible from a waypoint depends on whether or not a rock sample (intu- 
ition: a large piece of rock) is located at the waypoint. The probability of visibility is much 
higher if the latter is not the case. Specifically, the visibility of each objective depends on the 
locations of two rock samples, and if a rock sample is present, then the visibiUty probability 
drops to 0.1. 

• RoversPPP extends RoversPP by introducing the need to collect data about water existence. 
Each of the soil samples has a certain probability (< 1) to be "wet". For communicated 
sample data, an additional operator tests whether the sample was wet. If so, a fact "know- 
that-water" contained in the goal is set to true. The probabiUty of being wet depends on the 
location of the sample. 

We show no runtime plots for Logistics, Grid, and Rovers, since POND runs out of either time or 
memory on all considered instances of these domains. Table 5 shows that the scaling behavior of 
Probabilistic-FF in these three domains is similar to that observed in the previous domains. The 
goals in the RoversPPP problem cannot be achieved with probabiUties 6 G {0.75, 1.0}. This is 
proved by Probabilistic-FF's heuristic function, providing the correct answer in spUt seconds. 

5.2 Probabilistic Actions 

Our first two domains with probabilistic actions are the famous "Sand-Castle" (Majercik & Littman, 
1998) and "Slippery-Gripper" (Kushmerick et al., 1995) domains. The domains are simple, but they 
posed the first challenges for probabiUstic planners; our performance in these domains serves an 
indicator of the progress relative to previous ideas for probabilistic planning. 

In Sand-Castle, the states are specified by two boolean variables moat and castle, and state 
transitions are given by two actions dig-moat and erect-castle. The goal is to erect the castle. 
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(a) 



(b) 



Figure 18: Probabilistic-FF and POND on problems from (a) ID-WalkGrid with = 0.9, and (b) 
2D-WalkGrid with = 0.01. 



Building a moat with dig-m,oat might fail with probability 0.5. Erecting a castle with erect-castle 
succeeds with probability 0.67 if the moat has already been built, and with probability 0.25, other- 
wise. If failed, erect-castle also destroys the moat with probability 0.5. Figure 17(a) shows that 
both Probabilistic-FF and POND solve this problem in less than a second for arbitrary high values 
of 0, with the performance of both planners being almost independent of the required probability of 
success. 

Slippery-Gripper is already a bit more complicated domain. The states in Slippery-Gripper 
are specified by four boolean variables grip-dry, grip-dirty, block-painted, and block-held, and 
there are four actions dry, clean, paint, and pickup. In the initial state, the block is neither painted 
nor held, the gripper is clean, and the gripper is dry with probability 0.7. The goal is to have a 
clean gripper holding a painted block. Action dry dries the gripper with probability 0.8. Action 
clean cleans the gripper with probability 0.85. Action paint paints the block with probability 1, 
but makes the gripper dirty with probability 1 if the block was held, and with probability 0.1 if it 
was not. Action pickup picks up the block with probability 0.95 if the gripper is dry, and with 
probability 0.5 if the gripper is wet. 

Figure 17(b) depicts (on a log-scale) the relative performance of Probabilistic-FF and POND 
on Slippery-Gripper as a function of growing 0. The performance of ProbabiUstic-FF is nicely flat 
around 0.06 seconds. This time, the comparison with POND was somewhat problematic, because, 
for any fixed 0, POND on Shppery-Gripper exhibited a huge variance in runtime. In Figure 17(b) 
we plot the best runtimes for POND, as well as its average runtimes. The best run-times for POND 
for different values of vary around a couple of seconds, but the average runtimes are significantly 
worse. (For some high values of POND timed-out on some sample runs, and thus the plot provides 
a lower bound on the average runtimes.) 

In the next two domains, "ID-WalkGrid" and "2D-WalkGrid", the robot has to pre-plan a se- 
quence of conditional movements taking it from a comer of the grid to the farthest (from the initial 
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position) comer (Hyafil & Bacchus, 2004). In ID-WalkGrid the grid is one-dimensional, while 
in 2D-WalkGrid the grid is two-dimensional. Figure 18(a) depicts (on a log-scale) a snapshot of 
the relative performance of Probabilistic-FF and POND on one-dimensional grids of width n and 
= 0.9. The robot is initially at (1, 1), should get to (1, n), and it can try moving in each of the two 
possible directions. Each of the two movement actions moves the robot in the right direction with 
probability 0.8, and keeps it in place with probability 0.2. It is easy to see from Figure 18(a) that the 
difference between the two planners in this domain is substantial — while runtime of Probabilistic- 
FF grows only linearly with x, the same dependence for POND is seemingly exponential. 

The 2D-WalkGrid domain is already much more challenging for probabihstic planning. In all 
2D-WalkGrid problems with n x n grids the robot is initially at (1, 1), should get to {n, n), and it 
can try moving in each of the four possible directions. Each of the four movement actions advances 
the robot in the right direction with probability 0.8, in the opposite direction with probability 0, 
and in either of the other two directions with probability 0.1. Figure 18(a) depicts (on a log-scale) 
a snapshot of the relative performance of Probabilistic-FF and POND on 2D-WalkGrid with very 
low required probability of success 9 = 0.01, and this as a function of the grid's width n. The 
plot shows that ProbabiUstic-FF still scales well with increasing n (though not linearly anymore), 
while POND time-outs for all grid widths n > 3. For higher values of 9, however, Probabilistic-FF 
does reach the time-out limit on rather small grids, notably n = 6 and n = 5 for 6 = 0.25 and 
9 = 0.5, respectively. The reason for this is that Probabihstic-FF's heuristic function is not good 
enough at estimating how many times, at an early point in the plan, a probabilistic action must be 
applied in order to sufficiently support a high goal threshold at the end of the plan. We explain this 
phenomenon in more detail at the end of this section, where we find that it also appears in a variant 
of the well-known Logistics domain. 

Our last set of problems comes from the standard Logistics domain. Each problem instance 
x-y-z contains x locations per city, y cities, and z packages. We will see that Probabilistic-FF 
scales much worse, in Logistics, in the presence of probabilistic effects than if there is "only" initial 
state uncertainty (we will explain the reason for this at the end of this section). Hence we use much 
smaller instances than the ones used above in Section 5.1. Namely, to allow a direct comparison to 
previous results in this domain, we closely follow the specification of Hyafil and Bacchus (2004). 
We use instances with configurations x-y-z = 2-2-2, 4-2-2, and 2-2-4, and distinguish between two 
levels of uncertainty. 

• L-x-y-z correspond to problems with uncertainty only in the outcome of the load and unload 
actions. Specifically, the probabilities of success for load are 0.875 for trucks and 0.9 for 
airplanes, and for unload, 0.75 and 0.8, respectively. 

• LL-x-y-z extends L-x-y-z with independent uniform priors for each initial location of a 
package within its start city. 

Figure 19 depicts (on a log scale) runtimes of Probabilistic-FF and POND on L-2-2-2, L-4-2-2, 
and L-2-2-4, as a function of growing 9. On these problems, both planners appear to scale well, 
with the runtime of ProbabiUstic-FF and the optimal runtime of POND being roughly the same, 
and the average runtime of POND somewhat degrading from 2-2-2 to 4-2-2 to 2-2-4. This shows 
that both planners are much more efficient in this domain than the previously known SAT and CSP 
based techniques. However, moving to LL-x-y-z changes the picture for both planners. The results 
are as follows: 
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Figure 19: Probabilistic-FF and POND on problems from Logistics (a) L- 2-2-2, (b) L-4-2-2, and 
(c) L-2-2-4. 



1. On LL-2-2-2, the runtimes of Probabilistic-FF were identical to those on L-2-2-2, and the 
optimal runtimes of POND only slightly degraded to 2— 8 seconds. However, for all examined 
values of 9, some runs of POND resulted in timeouts. 

2. On LL-4-2-2, the runtimes of Probabilistic-FF were identical to those on L-4-2-2 for 9 G 
{0.01, 0.25, 0.5, 0.75}, yet Probabilistic-FF time-outed on 9 = 0.95. The optimal runtimes 
of POND degraded from those for L-4-2-2 only to 9 — 18 seconds, and again, for all values 
of 9, some runs of POND resulted in timeouts. 

3. On LL-2-2-4, Probabilistic-FF experienced hard times, finishing in 0.19 seconds for 9 = 
0.01, and time-outing for all other examined values of 9. The optimal runtimes of POND 
degraded from those for L-2-2-4 to 120 — 700 seconds, and here as well, for all values of 9, 
some runs of POND resulted in timeouts. 

We also tried a variant of LL-x-y-z with non-uniform priors over the initial locations of the pack- 
ages, but this resulted in a qualitatively similar picture of absolute and relative performance. 

The LL-x-y-z domain remains challenging, and deserves close attention in the future develop- 
ments for probabiUstic planning. In this context, it is interesting to have a close look at what the 
reasons for the failure of Probabilistic-FF is. It turns out that Probabilistic-FF is not good enough 
at estimating how many times, at an early point in the plan, a probabilistic action must be applied 
in order to sufficiently support a high goal threshold at the end of the plan. To make this concrete, 
consider a Logistics example with uncertain effects of all load and unload actions. Consider a pack- 
age P that must go from a city A to a city B. Let's say that P is initially not at A's airport. If the 
goal threshold is high, this means that, to be able to succeed, the package has to be brought to A's 
airport with a high probability before loading it onto an airplane. This is exactly the point where 
ProbabiUstic-FF's heuristic function fails. The relaxed plan contains too few actions unloading P 
at A's airport. The effect is that the search proceeds too quickly to loading P onto a plane and 
bringing it to B. Once the search gets to the point where B should be unloaded to its goal loca- 
tion, the goal threshold cannot be achieved no matter how many times one unloads P. At this point. 
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Probabilistic-FF's enforced hill-climbing enters a loop and eventually fails because the relaxed plan 
(which over-estimates the past achievements) becomes empty. 

The challenge here is to devise methods that are better at recognizing how many times P has 
to be unloaded at A's airport in order to sufficiently support the goal threshold. The error made by 
Probabilistic-FF lies in that our propagation of weights on the implication graph over-estimates the 
goal probability. Note here that this is much more critical for actions that must be applied early on 
in the plan, than for actions that are apphed later. If an action a appears early on in a plan, then 
the relaxed plan, when a is executed, will be long. Recall that the weight propagation proceeds 
backwards, from the goal towards the current state. At each single backwards step, the propagation 
makes an approximation that might lose precision of the results. Over several backwards steps, 
these imprecisions accumulate. Hence the quality of the approximation decreases quickly over the 
number of backwards steps. The longer the distance between goal and current state is, the more 
information is lost. We have observed this phenomenon in detailed experiments with different 
weight propagation schemes, that is, with different underlying assumptions. Of the propagation 
schemes we tried, the independence assumption, as presented in this paper, was by far the most 
accurate one. All other schemes failed to deliver good results even for much shorter distances 
between the goal and the current state. 

It is interesting to consider how this issue affects POND, which uses a very different method for 
estimating the probabiUty of goal achievement: instead of performing a backwards propagation and 
aggregation of weight values, POND sends a set of random particles through the relaxed planning 
graph in a forward fashion, and stops the graph building if enough particles end up in the goal. From 
our empirical results, it seems that this method suffers from similar difficulties as Probabilistic-FF, 
but not to such a large extent. POND's optimal runtimes for LL-x-y-z wee much higher than those 
for L-x-y-z. This indicates that it is always challenging for POND to "recognize" the need for 
applying some action a many times early on in the plan. More interestingly, POND never times-out 
in L-x-y-z, but it does often time-out in LL-x-y-z. This indicates that, to some extent, it is a matter 
of chance whether or not POND's random particles recognize the need for applying an action a 
many times early on in the plan. An intuitive explanation is that the "good cases" are those where 
sufficiently many of the particles failed to reach the goal due to taking the "wrong effect" of a. 
Based on this intuition, one would expect that it helps to increase the number of random particles in 
POND's heuristic function. We did so, running POND on LL-x-y-z with an increased number of 
particles, 200 and 600 instead of the default value of 64. To our surprise, the qualitative behavior of 
POND did not change, time-outing in a similar number of cases. It is unclear to us what the reason 
for this phenomenon is. Certainly, it can be observed that the situation encoded in LL-x-y-z is not 
solved to satisfaction by either of Probabilistic-FF's weight propagation or POND's random particle 
methods, in their current configurations. 

At the time of writing, it is unclear to the authors how better methods could be devised. It seems 
unlikely that a weight propagation - at least one that does not resort to expensive reasoning - exists 
which manages long distances better than the independence assumption. An alternative way out 
might be to simply define a weaker notion of plans that allows to repeat certain kinds of actions - 

18. This does not happen in the above L-2-2-2, L-4-2-2, and Z/-2-2-4 instances simply because they are too small and a 
high goal probability can be achieved without thinking too much about the above problem; if one increases the size 
of these instances, the problem appears. The problem appears earlier in the presence of initial state uncertainty - even 
in small instances such as LL-2-2-2, LL-4-2-2, and LL-2-2-4 - because with uncertainty about the start position of 
the packages one needs to try unloading them at the start airports more often. 
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throwing a dice or unloading a package - arbitrarily many times. However, since our assumption is 
that we do not have any observability during plan execution, when executing such a plan there would 
still arise the question how often an action should be tried. Since Logistics is a fairly well-solved 
domain in simpler formalisms - by virtue of Probabilistic-FF, even in the probabilistic setting as 
long as the effects are deterministic - we consider addressing this problem as a quite pressing open 
question. 

6. Conclusion 

We developed a probabilistic extension of Conformant-FF's search space representation, using 
a synergetic combination of Conformant-FF's SAT-based techniques with recent techniques for 
weighted model counting. We further provided an extension of conformant relaxed planning with 
approximate probabilistic reasoning. The resulting planner scales well on a range of benchmark do- 
mains. In particular it outperforms its only close relative, POND, by at least an order of magnitude 
in almost all of the cases we tried. 

While this point may be somewhat obvious, we would like to emphasize that our achievements 
do not solve the (this particular) problem once and for all. Probabilistic-FF inherits strengths and 
weaknesses from FF and Conformant-FF, like domains where FF's or Conformant-FF's heuristic 
functions yield bad estimates (e.g. the mentioned Cube-center variant). What's more, the proba- 
bilistic setting introduces several new potential impediments for FF's performance. For one thing, 
weighted model counting is inherently harder than SAT testing. Though this did not happen in our 
set of benchmarks, there are bound to be cases where the cost for exact model counting becomes 
prohibitive even in small examples. A promising way to address this issue lies in recent methods 
for approximate model counting (Gomes, Sabharwal, & Selman, 2006; Gomes, Hoffmann, Sabhar- 
wal, & Selman, 2007). Such methods are much more efficient than exact model counters. They 
provide high-confidence lower bounds on the number of models. The lower bounds can be used in 
Probabilistic-FF in place of the exact counts. It has been shown that good lower bounds with very 
high confidecne can be achieved quickly. The challenge here is to extend the methods - which are 
currently designed for non-weighted CNFs - to handle weighted model counting. 

More importantly perhaps, in the presence of probabilistic effects there is a fundamental weak- 
ness in Probabilistic-FF's - and POND's - heuristic information. This becomes a pitfall for perfor- 
mance even in a straightforward adaptation of the Logistics domain, which is otherwise very easy 
for this kind of planners. As outlined, the key problem is that, to obtain a high enough confidence 
of goal achievement, one may have to apply particular actions several times early on in the plan. 
Neither Probabilistic-FF's nor POND's heuristics are good enough at identifying how many times. 
In our view, finding techniques that address this issue is currently the most important open topic in 
this area. 

Apart from addressing the latter challenge, we intend to work towards applicability in real-word 
settings. Particularly, we will look at the space application settings that our Rovers domain hints at, 
at medication-type treatment planning domains, and at the power supply restoration domain (Bertoli, 
Cimatti, Slaney, & Thiebaux, 2002). 
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Appendix A. Proofs 

Proposition 2 Let (A, Mhj , G, 6) be a probabilistic planning problem described over k state vari- 
ables, and a be an m-step sequence of actions from A. Then, we have \Afba \ — 0{\J\fbj\+iTia{k+l)) 
where a is the largest description size of an action in A. 

Proof: The proof is rather straightforward, and it exploits the local structure of A/^j^'s CPTs. The 
first nodes/CPTs layer Af(o) of Mh^^ constitutes an exact copy of A4j • Then, for each 1 < i < m, the 
t-\h layer of Mb^ contains A; + 1 node {Y(^£^ } U X(^^^ . 

First, let us consider the "action node" Y(iy While specifying the CPT TY{t) in a straightforward 
manner as if prescribed by Eq. 4 might result in an exponential blow up, the same Eq. 4 suggests 
that the original description of a* is by itself a compact specification of TY{t)- Therefore, Tyit) 
can be described in space 0(a), and this description can be efficiently used for answering queries 
TY{t)(X{i) = £ I tt) as in Eq. 4. Next, consider the CPT Tx(t) of a state-variable node X{t) G X{fy 
This time, it is rather evident from Eq. 5 that Tx[t) can be described in space 0{a) so that queries 
Tx(t){X{t) = X I ^(t-i) = x') could be efficiently answered. Thus, summing up for all layers 
1 < i < m, the description size of \Nb^\ = 0{\Mbi I + 'm,a{k + 1)) ■ 

Lemma 4 Given a node v{t') € Imp_^p(j.y we have (^^(^0) = ^ ('"(^')) if and only if, given v 
at time t', the sequence of effects E{Imp,^,^^,-^^p^^-^) achieves p at t with probability 1. 

Proof: The proof of Lemma 4 is by a backward induction on the time layers of Imp^^^,^_^p^^y For 
time t, the only node of Jmp^p^j-) time-stamped with t is p{t) itself. For this node we do have 
lUp(t) {p{t)) = w {p{t)) = 1, but, given p at time t, an empty plan corresponding to (empty) 
E{Impp^^^^p^f--j ) trivially "re-establishes" p at t with certainty. Assuming now that the claim holds 
for all nodes of Jmp^^p^^^ time stamped with t' + 1, . . . ,t, v/e now show that it holds for the nodes 
time stamped with t'. 

It is easy to see that, for any node v{t') G Imp^pf^^y we get 'cOp^t) (^(*')) = ^ only if 

a goes down to zero. First, consider the chance nodes e{t') G Imp^^pf^^y For such a node, Ih is 
set to zero if and only if we have mp^i^ {r{t' + 1)) = 1 for some r G add{e). However, by our 
inductive assumption, in this and only in this case the effects E{lmp^(^^l-^^p^^J^-^-^) achieve pdXt with 
probabiUty 1, given the occurrence of e at time t'. 

Now, consider the fact nodes q{t') G Imp,^,^p(j-y For such a node, a can get nullified only by 
some effect e G E{a),a G A{t'),con{e) = q. The latter happens if only if, for all possible out- 
comes of e, (i) the node e{t') belongs to lmp^p^j.y and (ii) and the estimate vcip(^i^ (e(i')) = w{e{t')). 
In other words, by our inductive assumption, given any outcome e G A(e) at time t', the ef- 
fects E{Imp^^^,^^p(^f-^) achieve p at t with probability 1. Thus, given q at time t', the effects 
£'(Jmpy(j/-)^p(-j)) achieve p at t with probability 1 independently of the actual outcome of e. Al- 
ternatively, if for q{t') we have Ih > 0, then for each effect e conditioned on q{t), there exists an 
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outcome e of e such that, according to what we just proved for the chance nodes time-stamped with 
t', the effects E{Imp^(^^,^^JJ(^^_^_l^) do not achieve p at t with probability 1. Hence, the whole set of 
effects £'(Impg(j,)_j.p(j_,_^-)) does not achieve patt with probabihty 1. ■ 

Lemma 5 Let (A, Mbj , G, 6) be a probabilistic planning task, a be a sequence of actions applicable 
in bj, and \l be a relaxation function for A. For each time step t > —m, and each proposition p € 
V, if P{t) is constructed by build-PRPG(a, A, (j){J\fbj),G, 9, |^), then p at time t can be achieved 
by a relaxed plan starting with 

(1) with probability > f that is, p is not negatively known at time t) if and only ifp G uP(t) U P{t), 
and 

(2) with probability 1 (that is, p is known at time t) if and only ifp € P{t). 

Proof: The proof of the "if" direction is by a straightforward induction on t. For t = —m the claim 
is immediate by the direct initialization of uP{—m) and P{—m). Assume that, for —m < t' < t, 
if p G uP{t') U P{t'), then p is not negatively known at time t', and if p G P{t'), then p is known 
at time t'. 

First, consider some p{t) G uP{t) U P{t), and suppose that p is egatively know at time t. By 
the inductive assumption, and the property of the PRPG construction that uP{t — 1) U P(t — 1) C 
uP{t) U P{t), we have p ^ uP{t - 1) U P{t - 1). Therefore, p has to be added into uP{t) (and 
then, possibly, moved from there to P{t)) in the first for loop of the build-timestep procedure. 
However, if so, then there exists an action a G A{t — 1), e G E{a), and e G A(e) such that (i) 
con(e) G uP{t — 1) U P{t — 1), and (ii) p G add{e). Again, by the assumption of the induction we 
have that pre{a) is known at time t — 1, and con(e) is not negatively known at time t — 1. Hence, the 
non-zero probability of e occurring at time t implies that p can be achieved at time t with probability 
greater than 0, contradicting that p is negatively know at time t. 

Now, let us consider some pit) G P{t). Notice that, for t > —m, we have p{t) G P{t) if and 
only if 

V ^ ■ (28) 

Thus, for each world state w consistent with bj , we have either g G w for some fact proposition 
q{—m) G support{p{t)), or, for some effect e of an action a{t') G A{t'), t' < t, we have con{e) G 
P{t') and {e{t') \ e G A(e)} C support{p{t)). In this first case. Lemma 4 immediately implies that 
the concatenation of a]]*" with an arbitrary linearization of the (relaxed) actions ^(0), . . . , A{t — 1) 
achieves p att with probability 1, and thus p is known at time t. In the second case, our inductive 
assumption implies that con(e) is known at time t, and together with Lemma 4 this again imphes that 
the concatenation ofa\i with an arbitrary Unearization of the (relaxed) actions A(0), . . . , A(t — 1) 
achieves patt with probability 1. 

The proof of the "only if" direction is by induction on t as well. For t = —m this claim is 
again immediate by the direct initialization of P{—m). Assume that, for —m < t' < t, if p is not 
negatively known at time t', then p G uP{t') U P{t'), and if p is known at time t', then p G P{t'). 
First, suppose that p is not negatively known at time t, and yet we have p ^ uP{t) U P{t). From 
our inductive assumption plus that A{t — 1) containing all the NOOP actions for propositions in 
uP{t — 1) U P{t — 1), we know that p is negatively known at time t — 1. If so, then p can become 
not negatively known at time t only due to some e G A(e), e G E{a), such that pre{a) is known 
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at time t — I, and con(e) is not negatively known at time t — 1. By our inductive assumption, the 
latter conditions imply con(e) G uP{t — 1) U P{t — 1), and pre(a) G P{t — 1). But if so, then p 
has to be added to uP{t) U P{t) by the first for loop of the build-timestep procedure, contradicting 
our assumption that p ^ uP{t) U P{t). 

Now, let us consider some p known at time t. By our inductive assumption, P{t — 1) contains 
all the facts known at time t — 1, and thus A{t — I) is the maximal subset of actions A\f applicable 
at time t — 1. Let us begin with an exhaustive classification of the effects e of the actions A{t — 1) 
with respect to our p at time t. 

(I) Ve G A(e) : p G add{e), and con(e) G P{t - 1) 

(n) Ve G A(e) : p G add{e), and con(e) G uP{t - 1) 

(m) 3£ G A(e) : p ^ add{e) or con(e) P(i - 1) U uP{t - 1) 

If the set (I) is not empty, then, by the construction of build-w-impleafs(p(t), Imp), we have 

{£{t - 1) I £ G A(e)} C support{p{t)), 

for each e G (I). Likewise, by the construction of build-timestep (notably, by the update of for 
each e G (I), we have 

{e(t-l)\e€A(e)} 

Putting these two facts together, we have that Eq. 28 holds for p at time t, and thus we have p € P{t). 

Now, suppose that the set (I) is empty. It is not hard to verify that no subset of only effects (HI) 
makes p known at time t. Thus, the event "at least one of the effects (II) occurs" must hold with 
probability 1. First, by the construction of build-w-impleafs(p(i), Imp), we have 

support {p{t)) ^ support {con{e)(t — 1)) 
ee(n) 

Then, and 4 from Lemma 4 we have that the event "at least one of the effects (IT) occurs" holds with 
probabiUty 1 if and only if 

V ^ 

ee(n) 

I € support{con (e)(t— 1)) 

Putting these two facts together, we have that Eq. 28 holds for p at time t, and thus we have p e P{t). 
m 

Theorem 6 Let {A,J\fbj, G, 9) be a probabilistic planning task, a be a sequence of actions appli- 
cable in bj, and \^ be a relaxation function for A. If bu\\d-PRPG{a, A, <p[Afbj),G, 9, |^) returns 
FALSE, then there is no relaxed plan for {A, 6/, G, 9) that starts with a\i. 

Proof: Let t > be the last layer of the PRPG upon the termination of build-PRPG. For every 
—m < t' < t, by the construction of PRPG and Lemma 5, the sets P{t') and uP{t') contain all 
(and only all) propositions that are known (respectively unknown) after executing all the actions in 
the action layers up to and including A(t' — 1). 
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First, let us show that if build-PRPG returns FALSE, then the corresponding termination cri- 
terion would hold in all future iterations. If P{t + 1) = P{t), then we have A{t + 1) = A{t). 
Subsequently, since P{t + 1) U uP{t + 1) = P{t) U uP{t) and A{t + 1) = A{t), we have 
P{t + 2) U uP{t + 2) = P{t + 1) U uP{t + 1). Given that, we now show that P{t + 2) = P{t + 1) 
and uP{t + 2) = uP{t + 1). 

Assume to the contrary that there exists p{t + 2) G P{t + 2) such that p{t + 1)0 P{t + 1), that 
is p{t + 1) € uP{t + 1). By the construction of the sets P{t + 1) and P{t + 2) in the build-timestep 
procedure, we have 

V ^ ' 

lesupportipit+2)) ^^^^ 
i € support(p(t + 1 ) ) 

Consider an exhaustive classification of the effects e of the actions A{t + 1) with respect to our p at 
time t + 2. 

(I) V£ G A(e) : p e add{e), and con(e) G P{t + 1) 
(II) Ve G A(e) : p G add{e), and con(e) G uP{t + 1) 
(in) 3e G A(e) : p add{e) or con(e) P(t + 1) U uP{t + 1) 

Suppose that the set (I) is not empty, and let e G (I). From P{t) = P{t + 1) we have that con{e) G 
P{t), and thus {e{t) \ e G A(e)} C support{p{t + 1)). By the update of $ in build-timestep we 
then have $ V{£(t)|eeA(e)} and thus $ ^ Vig™pport(p(t+i)) ^' contradicting Eq. 29. 

Alternatively, assume that the set (I) is empty. Using the arguments similar to these in the proof 
of Lemma 5, p{t + 2) e P{t + 2) and p{t + 1) ^ P{t + 1) in this case imply that 



ee(n) 

lesupport{ccm{e){t+l)) ^^^^ 

$7^ V ^ 

ee(II) 
l^support{con(e)(t)) 

However, A{t + 1) = A{t), uP{t + 1) = uP{t), and P{t + 1) = P{t) together imply that all 
the action effects that can possibly take place at time t -I- 1 are also feasible to take place at time 
t. Therefore, since for each e G (11) we have con(e) G uP{t + 1) by the definition of (11), Eq. 30 
implies that 

[J support {con{e){t + 1)) n uP{-m) (J support {con{e){t)) r\uP{-m), (31) 
ee(ii) ee(n) 

contradicting our termination condition. Hence, we arrived into contradiction with our assumption 
thatp(i + l) + 

Having shown that P{t + 2) = P{t + 1) and uP{t + 2) = uP{t + 1), we now show that the 
termination criteria implies that, for each q{t -|- 2) G uP{t + 2), we have 

uP{—m) n support{p{t + 2)) = uP{—m) fl support{p{t + 1)). 
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Let -E'p(t+2) be the set of all effects of actions A{t + 1) such that con(e) G uP{t + 1), and, for each 
outcome e G A(e), we have p G add{e). Given that, we have 

uP(—m) n support{p{t + 2)) = uP{-m) n [J support(con{e){t + 1)) 

= uP{-m) n (J support{con{e){t)) , (32) 

= uP{—m) n support{p{t + 1)) 

where the first and third equalities are by the definition of support sets via Lemma 4, and the second 
equation is by our termination condition. 

The last things that remains to be shown is that our termination criteria implies get-P(i + 
2, G) =get-P(i + 1, G). Considering the simple cases first, if G g P{t + 1) U uP{t + 1), from 
P{t + 2)UuP{t + 2) = P{t + l)UuP{t + l) wehaveget-P(t + 2,G) =get-P(t + 1, G) = 0. Oth- 
erwise, if G C P(t + l),fromP(t + 2) = + 1) wehave get-P(t + 2, G) =get-P(t + 1, G) = 1. 

This leaves us with the case of G C P{t + 1) U uP{t + 1) and G n uP{t + 1) 7^ 0. From 
P(t + 2) = P{t + 1), uP{t + 2) = uP{t + 1), and the termination condition, we have 

G n uP{t) = G n uP{t + 1) = G n uP{t + 2). 

From get-P(t + 1, G) =get-P(t, G) we know that action effects that become feasible only in A{t) 
do not increase our estimate of probability of achieving any (/ € G fl uP{t + 1) from time t to time 
t + 1. However, from P{t + 1) = P{t), uP{t + 1) = uP{t), and A{t + 1) = A{t), we have that 
no action effect will become feasible at time t + 1 if it is not already feasible at time t, and thus 
get-P(t + 1, G) =get-P(t, G) will imply get-P(i + 2, G) =get-P(t + 1, G). 

To this point we have shown that if build-PRPG returns FALSE, then the corresponding ter- 
mination criterion would hold in all future iterations. Now, assume to the contrary to the claim of 
the theorem that build-PRPG returns FALSE at some iteration t, yet there exists a relaxed plan for 
{A, bj, G, 0) that starts with a\'^. First, if ^ = 1, then Lemma 5 imphes that there exists time T 
such that G C P{T). If so, then the persistence of our "negative" termination condition implies 
G C P{t). However, in this case we would have get-P(t, G) = 1 (see the second if of the get-P 
procedure), and thus build-PRPG would return TRUE before ever getting to check the "negative" 
termination condition in iteration t. Alternatively, if ^ = 0, then build-PRPG would have terminated 
with returning TRUE before the "negative" termination condition is checked even once. 

This leaves us with the case of < 6* < 1 and get-P(f,G) < 6. (get-P(t,G) > 6 will 
again contradict reaching the negative termination condition at iteration t.) We can also assume that 
G C Pit) U uP{t) because P{t) U uP{t) contains all the facts that are not negatively known at time 
t, and thus persistence of the negative termination condition together with G % P{t) U uP{t) would 
imply that there is no relaxed plan for any ^ > 0. Let us consider the sub-goals G n uP{t) ^ 0. 

(1) If for all subgoals g ^ G r\ uP{t), the implications in Imp^y(j-^ are only due to deterministic 
outcomes of the effects E{Imp^g(j^^), then the uncertainty about achieving G n uP{t) at time 
t is only due to the uncertainty about the initial state. Since the initial belief state is reasoned 
about with no relaxation, in this case get-P(t, G) = WMC($ A AgeG\P(t) ^9) provides us 
with an upper bound on the probabiUty of achieving our goal G by concatenated with 
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an arbitrary linearization of an arbitrary subset of ^(0), . . . , A{t — 1). The termination sub- 
condition get-P(t + 1, G) =get-P(t, G) and the persistence of the action sets A{T), T > t, 
imply then that get-P(i, G) provides us with an upper bound on the probabihty of achieving G 
by a\f concatenated with an arbitrary linearization of an arbitrary subset of A{0), . . . , A{T), 
for all T >t. Together with get-P(t, G) < 0, the latter conclusion contradicts our assumption 
that a desired relaxed plan exists. 

(2) If there exists a subgoal g e Gr\ uP{t) such that some impUcations in Imp^g^^-^ are due to truly 
probabilistic outcomes of the effects E{Imp^g(^^^), then repeating the (relaxed) actions A(t) in 
A{t + 1) will necessarily result in WMC($ A Ac,eG\P(t+i) 'fg) > WMC($ A AgeG\P(t) fg)' 
contradicting our termination sub-condition condition get-P(t + 1, G) =get-P(t, G). 

Hence, we arrived into contradiction that our assumption that build-PRPG returns FALSE at time t, 
yet there exists a relaxed plan for {A, bj, G, 9) that starts with a\l. ■ 
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